Code

WIP: Unicode Support#6748

Code

Closed

frabbit wants to merge 128 commits intoHaxeFoundation:developmentHaxeFoundation/haxe:developmentfrom frabbit:unicode_vectorfrabbit/haxe-1:unicode_vectorCopy head branch name to clipboard

Commits

Commits on Nov 1, 2017

• ucs2 implementation, work in progress

  frabbit

  committed
  1241b38

  Copy full SHA for 1241b38

• add unit tests for ucs2, utf8 and utf16 and make them pass

  frabbit

  committed
  251e189

  Copy full SHA for 251e189

• more work on utf16

  frabbit

  committed
  0d323df

  Copy full SHA for 0d323df

• [i18n] work on conversions, add more unit tests

  frabbit

  committed
  8f7f0b3

  Copy full SHA for 8f7f0b3

• add unused unit tests (2 folders for easier testing)

  frabbit

  committed
  febe263

  Copy full SHA for febe263

• only run i18n tests for now

  frabbit

  committed
  af88955

  Copy full SHA for af88955

• port convertUtf8ToUtf16 complete

  frabbit

  committed
  7913fe8

  Copy full SHA for 7913fe8

• get more unit tests working on js/python

  frabbit

  committed
  530b5e3

  Copy full SHA for 530b5e3

• add getCodeSize, was missing in last commit

  frabbit

  committed
  83b2e01

  Copy full SHA for 83b2e01

• get compilation working for java/cs/cpp (still failing unit tests)

  frabbit

  committed
  f46c1eb

  Copy full SHA for f46c1eb

• make sure the returned bytesdata length matches the bytes length

  frabbit

  committed
  d79b528

  Copy full SHA for d79b528

• get conversions between utf16 and utf8 on most platforms working

  frabbit

  committed
  29c0964

  Copy full SHA for 29c0964

• add temporary files and fixes to make testing and development of i18n easier

  frabbit

  committed
  00f474b

  Copy full SHA for 00f474b

• make more tests pass, fix some flash bugs

  frabbit

  committed
  c438968

  Copy full SHA for c438968

• get ucs2 tests working

  frabbit

  committed
  399d620

  Copy full SHA for 399d620

• fix Ucs2

  frabbit

  committed
  9cd7c22

  Copy full SHA for 9cd7c22

• get most unit tests working, some cleanup and refactoring

  frabbit

  committed
  d4a00f3

  Copy full SHA for d4a00f3

• get all string unit tests on most platforms working, further cleanup

  frabbit

  committed
  4ac9417

  Copy full SHA for 4ac9417

• further work on unicode, move utf8 related functions to Helper class, make abstracts simple (easier to deal with)

  frabbit

  committed
  f2046ac

  Copy full SHA for f2046ac

• prepare Utf8Tools for impl change

  frabbit

  committed
  6f376fe

  Copy full SHA for 6f376fe

• prepare impl change for utf8, add bytereader

  frabbit

  committed
  df9d7b1

  Copy full SHA for df9d7b1

• change impl of utf8, o(1) access to length

  frabbit

  committed
  a009b92

  Copy full SHA for a009b92

• more cleanup

  frabbit

  committed
  47231fc

  Copy full SHA for 47231fc

• use Utf8Tools without alias

  frabbit

  committed
  a4ec4d3

  Copy full SHA for a4ec4d3

• compile fix + cleanup

  frabbit

  committed
  f31a1ea

  Copy full SHA for f31a1ea

• work on utf16 refactoring

  frabbit

  committed
  60bd2dc

  Copy full SHA for 60bd2dc

• further work on utf16

  frabbit

  committed
  1b4fda3

  Copy full SHA for 1b4fda3

• change impl of Utf16 (make length o(1)), fix conversion tests

  frabbit

  committed
  d1f5141

  Copy full SHA for d1f5141

• add more inline methods to Utf16, start cleanup Ucs2 impl

  frabbit

  committed
  1972290

  Copy full SHA for 1972290

• get rid of EncodingTools, refactoring

  frabbit

  committed
  ad8fb89

  Copy full SHA for ad8fb89

• let Ucs2 Reader use the underlying string on platforms with native ucs2 encoding

  frabbit

  committed
  300f0ac

  Copy full SHA for 300f0ac

• small cleanup + utf16.compare optimizations

  frabbit

  committed
  ea359d0

  Copy full SHA for ea359d0

• faster utf8.compare

  frabbit

  committed
  5efa1bd

  Copy full SHA for 5efa1bd

• remove old utf8.compare

  frabbit

  committed
  a52b112

  Copy full SHA for a52b112

• fix more bugs, add more unit tests

  frabbit

  committed
  d48dc68

  Copy full SHA for d48dc68

• add a lot more tests, fix a lot of bugs

  frabbit

  committed
  5dacc05

  Copy full SHA for 5dacc05

• minor refactoring, use ByteAccess directly in ucs2.(toLowerCase|toUpperCase)

  frabbit

  committed
  2f0a57a

  Copy full SHA for 2f0a57a

• minor cleanup, make Ucs2Reader abstract

  frabbit

  committed
  810a7ff

  Copy full SHA for 810a7ff

• more tweaks, refactoring

  frabbit

  committed
  b7d9ae2

  Copy full SHA for b7d9ae2

• add basics for Utf32

  frabbit

  committed
  fb03dda

  Copy full SHA for fb03dda

• add setInt32 to js/ByteAccess

  frabbit

  committed
  ee21ea6

  Copy full SHA for ee21ea6

• whitespace fix

  frabbit

  committed
  bbed755

  Copy full SHA for bbed755

• work on conversions from and to utf32

  frabbit

  committed
  5ceddfc

  Copy full SHA for 5ceddfc

• cleanup

  frabbit

  committed
  a406209

  Copy full SHA for a406209

• cleanup + refactoring

  frabbit

  committed
  c7cf471

  Copy full SHA for c7cf471

• remove no_code_motion, add byteaccess compare and use it for all encodings, add helper functions to move bit shifting in one place

  frabbit

  committed
  f2a0af5

  Copy full SHA for f2a0af5

• remove some whitespace and unused commented out compare code

  frabbit

  committed
  df3a948

  Copy full SHA for df3a948

• small cleanup

  frabbit

  committed
  894eca2

  Copy full SHA for 894eca2

• make Uint8ArrayTools a private class in ByteAccess.js

  frabbit

  committed
  f1f9e4c

  Copy full SHA for f1f9e4c

• make BytesDataTools a private class in ByteAccess

  frabbit

  committed
  d6b9d30

  Copy full SHA for d6b9d30

• change order of classes, cleanup whitespace and imports

  frabbit

  committed
  609e0e2

  Copy full SHA for 609e0e2

• move Ucs2Reader and Ucs2Tools to Ucs2

  frabbit

  committed
  1831a9f

  Copy full SHA for 1831a9f

• move Utf8Tools and Utf8Reader to Utf8

  frabbit

  committed
  e50ac15

  Copy full SHA for e50ac15

• move Utf16Reader and Utf16Tools to Utf16

  frabbit

  committed
  9789d88

  Copy full SHA for 9789d88

• move Utf32Reader and Utf32Tools to Utf32

  frabbit

  committed
  bdfa134

  Copy full SHA for bdfa134

• move NativeStringTools to Tools, rename Encoding to Convert

  frabbit

  committed
  81a8ac5

  Copy full SHA for 81a8ac5

• fix cs bug (Convert is Encoding), small cleanup

  frabbit

  committed
  5d7f48c

  Copy full SHA for 5d7f48c

• use strToImplIndex instead of bit shift dircely

  frabbit

  committed
  db6ef9a

  Copy full SHA for db6ef9a

• add benchmarks for string functions

  frabbit

  committed
  1e114e5

  Copy full SHA for 1e114e5

• make Utf32.split and Ucs2.split a lot faster, add some inlines

  frabbit

  committed
  e66ad2e

  Copy full SHA for e66ad2e

• add 2 more split tests (see String.unit) and implement the special 'delimitier = empty string' case accordingly

  frabbit

  committed
  9bc72b4

  Copy full SHA for 9bc72b4

• make Tools classes private

  frabbit

  committed
  cda2858

  Copy full SHA for cda2858

• cleanup Benchmark and store test results to make sure that the code is not optimized away

  frabbit

  committed
  3b00e11

  Copy full SHA for 3b00e11

• update hl Bytes to match current hl structure

  frabbit

  committed
  e85e1c0

  Copy full SHA for e85e1c0

• improve benchmark code, add fastCodeAt for all encodings (at least temporary for utf8 and utf16)

  frabbit

  committed
  887e769

  Copy full SHA for 887e769

• use helper function for toLowerCase and toUpperCase to move iteration logic in one place

  frabbit

  committed
  d6a697d

  Copy full SHA for d6a697d

• fix for php7

  frabbit

  committed
  88d5b02

  Copy full SHA for 88d5b02

• add benchmark for php7, use luajit for lua

  frabbit

  committed
  d2c0780

  Copy full SHA for d2c0780

• use fastCodeAt in toBytes

  frabbit

  committed
  857d007

  Copy full SHA for 857d007

• add test for php7

  frabbit

  committed
  c4c7377

  Copy full SHA for c4c7377

• fix flash bug

  frabbit

  committed
  c7671e0

  Copy full SHA for c7671e0

• use Vector as underlying type for UTF32

  frabbit

  committed
  afe89df

  Copy full SHA for afe89df

• fix Check file

  frabbit

  committed
  c927bb4

  Copy full SHA for c927bb4

• add benchmarks for allocation (nativestring -> encoding)

  frabbit

  committed
  b342275

  Copy full SHA for b342275

• improve allocation of Utf32, use Utf16 allocation for Ucs2 (bytes are the same anyway)

  frabbit

  committed
  34dfa8a

  Copy full SHA for 34dfa8a

• replace strict flag enum with bool, improve eval performance

  frabbit

  committed
  4b15533

  Copy full SHA for 4b15533

• improve Benchmarks

  frabbit

  committed
  4c52f32

  Copy full SHA for 4c52f32

• further optimizations on unicode, still needs a lot of cleanup

  frabbit

  committed
  d0a4a43

  Copy full SHA for d0a4a43

• use relative paths for haxe executable

  frabbit

  committed
  be0d62d

  Copy full SHA for be0d62d

• further work on unicode performance

  frabbit

  committed
  3b03347

  Copy full SHA for 3b03347

• unify names of wrap/unwrap functions in ByteAccess

  frabbit

  committed
  189fc77

  Copy full SHA for 189fc77

• do more inlining in js

  frabbit

  committed
  07d1db9

  Copy full SHA for 07d1db9

• cleanup, directly add bytesData to buffer without creating Bytes

  frabbit

  committed
  e104c62

  Copy full SHA for e104c62

• [unicode] major cleanup, minor improvements + fixes

  frabbit

  committed
  b0d5548

  Copy full SHA for b0d5548

• [unicode] more cleanup, adjust licence

  frabbit

  committed
  467a348

  Copy full SHA for 467a348

• [unicode] clearer separation of charCodes (codePoints) and codeUnits, fixes

  frabbit

  committed
  25a4417

  Copy full SHA for 25a4417

• use codePoint wording to differentiate between code units and code points

  frabbit

  committed
  47c165a

  Copy full SHA for 47c165a

• minor adjustments for benchmarks, remove eval from Makefile (it's interp)

  frabbit

  committed
  2bb6e2e

  Copy full SHA for 2bb6e2e

• add benchmarks for charAt

  frabbit

  committed
  a7992c7

  Copy full SHA for a7992c7

• add optimizations for Utf8 and Utf16 when we know that we only have same sized characters

  frabbit

  committed
  e8201f4

  Copy full SHA for e8201f4

• remove comment

  frabbit

  committed
  9240c23

  Copy full SHA for 9240c23

• use native string for utf32 impl on python

  frabbit

  committed
  bc5d55e

  Copy full SHA for bc5d55e

• improve conversions from python utf32 to other encodings

  frabbit

  committed
  d3633e5

  Copy full SHA for d3633e5

• move implementation of to... functions to the target itself, allows better optimizations, remove some itermediate data structures for some conversations

  frabbit

  committed
  df160d3

  Copy full SHA for df160d3

• add benchmarks for conversions

  frabbit

  committed
  e686f6d

  Copy full SHA for e686f6d

• add lua to test-all

  frabbit

  committed
  8434d38

  Copy full SHA for 8434d38

• cleanup tests, add tests + bugfix for convertions from/to nativestrings

  frabbit

  committed
  072619d

  Copy full SHA for 072619d

• add substring optimizations for utf8 and utf16 when we have a fixed character size

  frabbit

  committed
  845ab1b

  Copy full SHA for 845ab1b

• use faster StringBuffer to create Ucs2 from ByteAccess

  frabbit

  committed
  cdccceb

  Copy full SHA for cdccceb

• utf16 is now based on string on ucs2 native platforms like (js, java, cs, hl), pull development branch

  frabbit

  committed
  0dd6b30

  Copy full SHA for 0dd6b30

• only check hl interp for now

  frabbit

  committed
  5c46681

  Copy full SHA for 5c46681

• add iterators for ucs2, utf16, utf8 and utf32

  frabbit

  committed
  324e3c2

  Copy full SHA for 324e3c2

• use little endian instead of big endian for decoding

  frabbit

  committed
  59383ae

  Copy full SHA for 59383ae

• add more tests (conversion to/from bytes, to/from native string

  frabbit

  committed
  0f4dc4e

  Copy full SHA for 0f4dc4e

Commits on Nov 2, 2017

• improve addBytesData, a little bit cleaner/more type safe

  frabbit

  committed
  08ca952

  Copy full SHA for 08ca952

• minor cleanup

  frabbit

  committed
  fcee5ca

  Copy full SHA for fcee5ca

• improve iterator code, use iterators to implement eachCode

  frabbit

  committed
  7d48eaa

  Copy full SHA for 7d48eaa

• never read single bytes in utf16, ucs2 or utf32, always read blocks of 2 (ucs2, utf16) or 4 bytes (utf32), prepare switching to little endian for internal bytes encoding

  frabbit

  committed
  27a9d58

  Copy full SHA for 27a9d58

• use little endian for underlying byte-based encodings and also for conversions like toBytes, fromBytes always expects little endian encoding

  frabbit

  committed
  9c76aed

  Copy full SHA for 9c76aed

• add more tests for toBytes

  frabbit

  committed
  84a151b

  Copy full SHA for 84a151b

• cleanup, split Utf32 into two files

  frabbit

  committed
  d6db4fb

  Copy full SHA for d6db4fb

• minor cleanup

  frabbit

  committed
  13e8cc9

  Copy full SHA for 13e8cc9

• try fix cs compile error

  frabbit

  committed
  b3fc1c9

  Copy full SHA for b3fc1c9

• add i18n classes to host for cppia to fix unit tests (Unknown function : __hxcpp_string_of_bytes(4))

  frabbit

  committed
  a39aacc

  Copy full SHA for a39aacc

• also add ByteAccessBuffer to hostclasses

  frabbit

  committed
  cd3881c

  Copy full SHA for cd3881c

• uncomment big endian functions for now

  frabbit

  committed
  ce9aaaf

  Copy full SHA for ce9aaaf

Commits on Nov 20, 2017

• move i18n bench test to tests/benchs/i18n

  frabbit

  committed
  0c43003

  Copy full SHA for 0c43003

• uncomment all tests unrelated to i18n

  frabbit

  committed
  fd0d0cd

  Copy full SHA for fd0d0cd

• Merge remote-tracking branch 'origin/development' into unicode_vector

  frabbit

  committed
  2decaa1

  Copy full SHA for 2decaa1

• support same filenames in different packages like Utf8.unit.hx and haxe/i18n/Utf8.unit.hx

  frabbit

  committed
  6422ea1

  Copy full SHA for 6422ea1

• only call eqAbstract if both types are the same

  frabbit

  committed
  2c83595

  Copy full SHA for 2c83595

• remove some unused functions

  frabbit

  committed
  7460490

  Copy full SHA for 7460490

• Move BytesDataTools into it's own Module, solves cppia issue with the call to __hxcpp_string_of_bytes, see HaxeFoundation/hxcpp#669

  frabbit

  committed
  b7d3e72

  Copy full SHA for b7d3e72

• hide functions for js, should fix dox generation

  frabbit

  committed
  ef89692

  Copy full SHA for ef89692

Commits on Nov 21, 2017

• include impl classes, hopefully fix cppia unit test compilation error

  frabbit

  committed
  dd79bc7

  Copy full SHA for dd79bc7

• fix stupid bug and remove trace

  frabbit

  committed
  a672a3a

  Copy full SHA for a672a3a

Commits on Nov 22, 2017

• Merge remote-tracking branch 'origin/development' into unicode_vector

  frabbit

  committed
  3472d6f

  Copy full SHA for 3472d6f

Commits on Apr 17, 2018

• Merge branch 'development' into unicode_vector

  Simn

  authored
  669b829

  Copy full SHA for 669b829