Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add functions to the language #704

Closed
wants to merge 219 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
219 commits
Select commit Hold shift + click to select a range
b27b809
lang: funcs: Remove old engine code
purpleidea Aug 9, 2023
e27c161
lang: interfaces: Partially change Graph signature
purpleidea Aug 11, 2023
8c4ffe6
lang: ast, interfaces: Plumb in the Graph sig changes and value pointers
purpleidea Aug 11, 2023
b91452d
lang: ast, interfaces, interpret: Change the Output sig
purpleidea Aug 11, 2023
e80d69d
lang: Update the Build signature to return a type
purpleidea Aug 9, 2023
c121c77
lang: funcs: Add listlookup function
purpleidea Aug 28, 2023
4ce816a
lang: interfaces, funcs: Add a new graph engine called dage
purpleidea Sep 4, 2023
05c90a5
lang: Add a new DAG function engine (dage)
purpleidea Aug 11, 2023
c2920a7
XXX bugs/hacking
purpleidea Aug 11, 2023
bfc4703
lang: ast, interfaces: Pass through graph environment
purpleidea Mar 4, 2023
9e9e5e9
lang: ast, interfaces, interpret: Pass through table
purpleidea Apr 23, 2023
9627b00
lang: interfaces: Remove Func signature
purpleidea May 3, 2023
f0a0d5e
lang: interfaces, funcs: Port Func API to new Stream signature
purpleidea May 28, 2023
5fb3d0b
lang: funcs: structs: Change names for tests
purpleidea May 7, 2023
f703872
XXX: This is a combination of 58 commits.
gelisam Nov 20, 2022
69c54e5
XXX: WIP FIXES
purpleidea Apr 25, 2023
a43284e
use FuncEdge, not SimpleEdge
gelisam May 7, 2023
d07a8b2
no need to call Function() a second time
gelisam May 7, 2023
9472378
[TMP] simplify test case
gelisam May 7, 2023
36ad945
[TMP] hush panic in .String()
gelisam May 7, 2023
ee1c7b1
[TMP] simplify test case
gelisam May 7, 2023
dc7dd96
[TMP] simplify the test case
gelisam May 7, 2023
4ef9170
SimpleFn now implements Value
gelisam May 7, 2023
04bd316
don't hardcode edge names
gelisam May 7, 2023
7bb18b4
use the right edge name for CallFunc
gelisam May 7, 2023
5eab18d
wrap func declarations in a ConstFunc
gelisam May 13, 2023
1f4fea9
ChannelBasedSink now sends values downstream
gelisam May 17, 2023
18d8e80
XXXDAGEXXX: temp graphviz debugging on Txn Commit() for Sam
purpleidea May 15, 2023
2e44f21
use the arg names from the type, not "arg%d XXX"
gelisam May 20, 2023
77bbd02
include the arg name in Info.Sig
gelisam May 20, 2023
94eac78
ChannelBasedSink should send the input value downstream, not the reco…
gelisam May 20, 2023
df0e282
don't busy loop on a closed channel
gelisam May 20, 2023
6500f95
XXX: port these functions to new stream API
purpleidea May 28, 2023
415d0d9
better names for a better graphviz visualization
gelisam May 20, 2023
265745d
Revert "[TMP] simplify test case"
gelisam May 20, 2023
258a5b9
remove unused Expr::Func() definitions
gelisam Jun 11, 2023
b401dcd
[TMP] simplify example
gelisam Jun 11, 2023
e8a4a5d
ExprParam
gelisam Jun 11, 2023
a7c6bef
ExprVar makes a copy of top-level defs
gelisam Jun 11, 2023
57e832e
scope-check the expression to which the ExprVar refers
gelisam Jun 11, 2023
edf4db2
ExprRecur, a fake Expr to detect recursion
gelisam Jun 11, 2023
82955dc
detect recursion
gelisam Jun 11, 2023
6eb53cd
drop outdated Scope.Indexes
gelisam Jun 11, 2023
217957c
ExprCall makes a copy of top-level defs
gelisam Jun 11, 2023
edb32a8
add context parameter to SetScope
gelisam Jun 11, 2023
40e04fe
add parameters to the context
gelisam Jun 11, 2023
d19df21
ExprVar can refer to a function parameter
gelisam Jun 11, 2023
742d21b
ExprCall can refer to a function parameter
gelisam Jun 11, 2023
af29a36
VarExpr is responsible for unifying its copy of the target
gelisam Jun 11, 2023
c871581
nuking nonsense section
gelisam Jun 11, 2023
057336c
[DEBUG] debug SetScope
gelisam Jun 20, 2023
69b43d2
[TMP] simplify example
gelisam Jun 20, 2023
5a2564b
ExprCall should use FuncInvariant, not CallInvariant
gelisam Jun 24, 2023
1296dbe
ExprParam's type comes from its ExprFunc
gelisam Jun 24, 2023
addbe88
[DEBUG] a different example
gelisam Jun 24, 2023
52da720
don't add top-level binds to the function graph
gelisam Jun 24, 2023
8fa391f
ExprCall should not use the environment, only params
gelisam Jun 25, 2023
7b2dbe1
Revert "[DEBUG] a different example"
gelisam Jun 25, 2023
7f781d0
Revert "[TMP] simplify example"
gelisam Jun 25, 2023
44410c2
Revert "[TMP] simplify example"
gelisam Jun 25, 2023
955fdb5
[TMP] simplify example
gelisam Jun 25, 2023
678a319
guard against i changing
gelisam Jun 25, 2023
a09ad12
add more nodes to SetScopeGraphviz
gelisam Jun 25, 2023
67db300
xmap
gelisam Jun 25, 2023
441de21
try to use the correct arg names
gelisam Jun 25, 2023
c092ef8
Revert "xmap"
gelisam Jun 25, 2023
402d7b4
[DEBUG] testing map
gelisam Jun 25, 2023
f1286b7
use correct arg name for ChannelBasedSink
gelisam Jun 25, 2023
96f4260
Revert "[DEBUG] testing map"
gelisam Jun 25, 2023
698d912
[DEBUG] test with map
gelisam Jun 25, 2023
8dee7e9
clarify where the 'no value in table' error comes from
gelisam Jun 25, 2023
f3f0bc1
copy ExprCall.expr before ExprCall.Graph()
gelisam Jun 25, 2023
532e33e
restore the type we previously set in SetType()
gelisam Jun 25, 2023
31efcf7
MapFunc should not remove subgraphInput...
gelisam Jun 27, 2023
5f925d5
XXXDAGEXXX: DEBUG Run graphviz a lot
purpleidea Jun 27, 2023
f797287
XXX: isn't Type() a good alternative to inspecting the function?
purpleidea Jun 27, 2023
2b69179
Fix one bug, and always send in case Cmp doesn't work here...
purpleidea Jun 27, 2023
b99be80
using new Txn.Erase() instead of double-commit
gelisam Jun 28, 2023
e8d4736
[DEBUG] simpler addition example
gelisam Jun 28, 2023
d4dd2aa
I have indeed changed my mind
gelisam Jun 28, 2023
bde2168
XXXDAGEXXX explain why ftyp is wrong but sometimes works
gelisam Jun 28, 2023
58ffb1c
lang: funcs: core: iter: Refactor map arg names
purpleidea Jun 29, 2023
97a17f4
lang: Update the Build signature to return a type
purpleidea Jun 29, 2023
727788b
Make this code more idiomatic
purpleidea Jun 29, 2023
8518569
xxx misc fixup
purpleidea Jun 30, 2023
232819f
XXX: add debugging
purpleidea Jun 29, 2023
5f6ff8f
yes, FuncValue.Call() should use the same Txn
gelisam Jun 30, 2023
f698230
use ftyp now that it is reliable
gelisam Jun 30, 2023
c3391a9
XXX: refactor map_func
purpleidea Jun 30, 2023
a87b4d5
XXXDAGEXXX: improve graphviz debugging
purpleidea Jun 30, 2023
b3ce422
XXXDAGEXXX add example-specific logging to the engine
gelisam Jun 30, 2023
70a0936
XXX: found some races. We can mutex this tomorrow
purpleidea Jun 30, 2023
4a3b6e6
I don't think we need to Copy here
gelisam Jun 30, 2023
74c48f0
[DEBUG] use the result of + in the test output
gelisam Jun 30, 2023
b063bd9
[DEBUG] look at map again
gelisam Jun 30, 2023
2763c83
XXXDAGEXXX XXX: avoid locks in safelogf
purpleidea Jun 30, 2023
d38adbc
Refactor type building
purpleidea Jun 30, 2023
630b080
Revert "I don't think we need to Copy here"
gelisam Jun 30, 2023
43cf14a
explain why we need the Copy().
gelisam Jun 30, 2023
2a241ce
XXXDAGEXXX reword inaccurate comment
gelisam Jun 30, 2023
c927e93
Revert "[DEBUG] look at map again"
gelisam Jun 30, 2023
548a9cb
restore original func-math1 test
gelisam Jun 30, 2023
c988f2c
XXXDAGEXXX Revert "add example-specific logging to the engine"
gelisam Jun 30, 2023
53c2d66
remove logging we no longer need
gelisam Jun 30, 2023
06c6732
[DEBUG] simplify the test case
gelisam Jun 30, 2023
d5a1083
fixed a bug, found another one
gelisam Jun 30, 2023
eb2f976
XXX: remove debugging
purpleidea Jul 3, 2023
39a6bbd
XXXDAGEXXX lang: funcs: Ensure we only send complete structs
purpleidea Jul 3, 2023
0554f16
XXXDAGEXXX XXX: WIP on new dage, split off old dage into pdage
purpleidea Jul 10, 2023
f5cfe06
XXXDAGEXXX XXX WIP new dage
purpleidea Jul 13, 2023
25d9e8f
gofmt fixes
purpleidea Aug 20, 2023
7f77fa6
XXXDAGEXXX XXX horrendous debugging misc
purpleidea Jul 15, 2023
50f5106
XXXDAGEXXX XXX: Txn changes to group in Reverse stuff with one commit
purpleidea Jul 15, 2023
3487e82
XXX: this roughly works
purpleidea Jul 15, 2023
a1741e0
XXXDAGEXXX XXX: improve naming of dage graph mutex
purpleidea Jul 16, 2023
f9a0697
XXX 100 ms
purpleidea Jul 16, 2023
70b07e9
XXXDAGEXXX XXX: misc txn flag stuff
purpleidea Jul 16, 2023
fbc8542
XXXDAGEXXX XXX woops
purpleidea Jul 16, 2023
3c82fda
XXXDAGEXXX XXX: perform reverse ops in reverse order
purpleidea Jul 16, 2023
8859424
XXX: pass through a name to make our graphs nicer
purpleidea Jul 16, 2023
80e8aa0
XXX: in a rare case of inference instead of knowing...
purpleidea Jul 16, 2023
89e8ca2
XXXDAGEXXX Fix small shutdown bug... What's left?
purpleidea Jul 16, 2023
311a84a
XXXDAGEXXX XXX GRAPHVIZ leftovers
gelisam Jul 16, 2023
b8bdce9
XXXDAGEXXX a bit more logging
gelisam Jul 16, 2023
84bde2a
add vertices before edges
gelisam Jul 18, 2023
7069ab6
XXXDAGEXXX XXX logging cleanup
purpleidea Jul 19, 2023
ab001f9
XXXDAGEXXX XXX clearer graphviz all in the same folder
purpleidea Jul 19, 2023
d5352be
XXXDAGEXXX XXX: cleanly catch any panics in stream
purpleidea Jul 19, 2023
533759a
XXXDAGEXXX XXX: improved engine, but not working perfectly.
purpleidea Jul 24, 2023
61bdade
XXXDAGEXXX dashed edge for ChannelBasedSourceFunc
gelisam Jul 25, 2023
3ecb51d
XXXDAGEXXX XXX: defer Wait from deleteVertex into main loop
purpleidea Jul 25, 2023
f220046
XXXDAGEXXX XXX stats improvements
purpleidea Jul 26, 2023
f2313cc
XXXDAGEXXX XXX more stats
purpleidea Jul 26, 2023
686f6d8
XXX: new test case for us to look at
purpleidea Jul 26, 2023
c5eadf0
XXXDAGEXXX XXX basic graph engine works fast!
purpleidea Jul 27, 2023
a1456ce
XXX: seeing if this fixes the txn issue
purpleidea Jul 27, 2023
c277294
XXX fix data race on concurrent copy
purpleidea Jul 27, 2023
4917cfd
XXX: add more error handling
purpleidea Jul 27, 2023
d95e553
propagate errors from Reverse
gelisam Jul 30, 2023
c0fe248
XXXDAGEXXX XXX: don't wait before resuming
purpleidea Jul 30, 2023
80a948a
XXXDAGEXXX XXXjamescomments
purpleidea Jul 30, 2023
26036d4
XXX: make the race actually readable
purpleidea Jul 30, 2023
7124d7e
XXX We shouldn't be writing here, it's a Copy
purpleidea Jul 30, 2023
b063f86
more examples
gelisam Jul 30, 2023
5dfd0a4
the function graph must contain Funcs, not Exprs
gelisam Jul 30, 2023
f4e1560
XXX: seems like we should probably check nil
purpleidea Jul 30, 2023
1f65d18
Expr.Graph() now returns the output node
gelisam Jul 30, 2023
fd61f14
XXXDAGEXXX XXX: first implementation of refcounting
purpleidea Aug 4, 2023
00763e4
XXXDAGEXXX XXX cleanup debugging slightly
purpleidea Aug 4, 2023
e93a41b
XXXDAGEXXX XXX use a Txn to add the initial graph
purpleidea Aug 4, 2023
f811b63
XXXDAGEXXX XXX start porting lang/lang to the new API
purpleidea Aug 4, 2023
cfb627b
xxx more map tests
purpleidea Aug 4, 2023
3a8ae5c
XXXDAGEXXX XXX improve performance substantially
purpleidea Aug 4, 2023
aae077f
XXX wait for shutdown to complete if we're measuring graph size == 0
purpleidea Aug 4, 2023
ced8742
XXX cache map inputs
purpleidea Aug 4, 2023
cbcc2fa
XXX improve tests
purpleidea Aug 4, 2023
6c10cbf
XXXDAGEXXX XXX refcount works with $x + $x example, but i think not c…
purpleidea Aug 6, 2023
23a0680
XXXDAGEXXX XXX better refcount... getting closer but...
purpleidea Aug 6, 2023
c0a1cde
XXXDAGEXXX XXX woops i am dumb, problem fixed
purpleidea Aug 6, 2023
d0ab20e
XXXDAGEXXX XXX cleaned up implementation of the reference counting
purpleidea Aug 6, 2023
4bb282c
XXXDAGEXXX XXX cleanup unnecessary debug noise and misc
purpleidea Aug 6, 2023
aeae735
XXX: misc test fixes
purpleidea Aug 6, 2023
08fec75
XXXDAGEXXX XXX: even on error or panic, the engine should shutdown cl…
purpleidea Aug 7, 2023
5db3343
XXXDAGEXXX XXX improve wake() debugging
purpleidea Aug 7, 2023
0c7b382
XXXDAGEXXX XXX drain the ag channel
purpleidea Aug 7, 2023
f4f69d9
XXXDAGEXXX XXX: dage: try and remove the default
purpleidea Aug 7, 2023
c1618f3
XXX better testing
purpleidea Aug 8, 2023
928d898
XXX: lang: Run the Reverse and Free before engine exits
purpleidea Aug 8, 2023
7a3ae06
XXX remove useless errwraps
purpleidea Aug 11, 2023
74a23a0
XXX graph edge was backwards
purpleidea Aug 11, 2023
a186e2c
XXXDAGEXXX XXX: lang: remove AddReverse which is useless and add AddG…
purpleidea Aug 11, 2023
e0bd6ac
XXXDAGEXXX XXX dage fixes for live functions
purpleidea Aug 13, 2023
85fd0ac
XXX more debug
purpleidea Aug 13, 2023
cec8efb
XXX fixup after rebase
purpleidea Aug 13, 2023
3b23aba
XXX: fix lexer/parser issues
purpleidea Aug 13, 2023
bfcfb55
XXX unused
purpleidea Aug 15, 2023
0ea8955
XXX: overwrite variant type
purpleidea Aug 22, 2023
5c1f4cf
XXX add more tests
purpleidea Aug 22, 2023
bf8d87e
lang: types: Avoid a panic if it's not settable
purpleidea Aug 22, 2023
e513be3
lang: types: Loop in deterministic order
purpleidea Aug 22, 2023
823d552
remember to fix this scoping bug
gelisam Aug 23, 2023
852d111
ExprPoly
gelisam Aug 24, 2023
6acd195
scope-check the body of a class in StmtInclude, not StmtClass
gelisam Aug 25, 2023
2be327a
restore previous error message
gelisam Aug 27, 2023
ec0d4b6
XXX: catch nil obj.function
purpleidea Aug 27, 2023
ccc8dac
XXX: iter.map arg order was backwards
purpleidea Aug 28, 2023
dbe54bc
Stmt.Graph() should not take an env
gelisam Aug 29, 2023
e608ef1
XXX: fix test as per sam comment (needs an XXX edit)
purpleidea Aug 29, 2023
9879d54
fix type signature in very-complex-example.txtar
gelisam Aug 29, 2023
d01312f
lang: funcs: dage: Be more quiet
purpleidea Aug 31, 2023
168d597
XXXDAGEXXX cleanup dage code. txn_test needs fixing for GC
purpleidea Sep 4, 2023
f9132b4
XXX: move the sink/source/const/call funcs into their own package
purpleidea Sep 5, 2023
fa590af
XXX: also move the earlier structs into their own package
purpleidea Sep 5, 2023
a501a97
XXX: rename fancyfunc (other names welcome?)
purpleidea Sep 5, 2023
361d318
lang: types: Export the base value for others
purpleidea Sep 5, 2023
383f94e
XXX: move these into a separate file and use the std edge
purpleidea Sep 5, 2023
c096d16
XXX: rename simplefn back to funcvalue for now
purpleidea Sep 5, 2023
06dd554
XXX: put these examples back, no reason to delete
purpleidea Sep 5, 2023
4ade499
XXX: misc cleanups
purpleidea Sep 5, 2023
b6a0b80
XXX: inline code
purpleidea Sep 5, 2023
07db900
typo
purpleidea Sep 18, 2023
1c15fa7
XXX: scopegraph
purpleidea Sep 18, 2023
5ab4724
XXX misc
purpleidea Sep 18, 2023
a0a2cef
XXX remove pdage for now
purpleidea Sep 18, 2023
77f2842
XXX: misc cleanups
purpleidea Sep 18, 2023
57c6530
XXX: fix old unification test, move it to new system
purpleidea Sep 19, 2023
5926dfc
XXX: misc tests port sort of
purpleidea Sep 19, 2023
edb74e6
XXX: empty graphs are allowed too
purpleidea Sep 19, 2023
721504c
XXX fix most tests except...
purpleidea Sep 19, 2023
2734547
XXX port more tests
purpleidea Sep 19, 2023
61e7d29
XXX add missing refcount
purpleidea Sep 19, 2023
6f99b1d
XXX most tests needed this
purpleidea Sep 19, 2023
e071fdb
XXX fix last test
purpleidea Sep 19, 2023
1c4ccc9
XXX txn tests
purpleidea Sep 19, 2023
f2f97d0
XXX more txn tests
purpleidea Sep 19, 2023
b09a5de
XXX when did i forget to have this here?
purpleidea Sep 19, 2023
b058ed9
XXX: golint fixes
purpleidea Sep 19, 2023
3434bc6
clearer ExprRecur errors
gelisam Sep 20, 2023
25eb138
drop ExprRecur
gelisam Sep 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ Similar logic applies for situations when you only specify the `mode` parameter.
This all turns out to be more safe and "correct", in that it would error and
prevent masking an error for a situation when you expected a file to already be
at that location. It also turns out to simplify the internals significantly, and
remove an ambiguous scenario with the reversable file resource.
remove an ambiguous scenario with the reversible file resource.

### Why do function names inside of templates include underscores?

Expand Down
9 changes: 4 additions & 5 deletions docs/language-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -563,21 +563,20 @@ would like to propose a more logical or performant variant.

#### Function graph generation

At this point we have a fully type AST. The AST must now be transformed into a
At this point we have a fully typed AST. The AST must now be transformed into a
directed, acyclic graph (DAG) data structure that represents the flow of data as
necessary for everything to be reactive. Note that this graph is *different*
from the resource graph which is produced and sent to the engine. It is just a
coincidence that both happen to be DAG's. (You don't freak out when you see a
coincidence that both happen to be DAG's. (You aren't surprised when you see a
list data structure show up in more than one place, do you?)

To produce this graph, each node has a `Graph` method which it can call. This
starts at the top most node, and is called down through the AST. The edges in
the graphs must represent the individual expression values which are passed
from node to node. The names of the edges must match the function type argument
names which are used in the definition of the corresponding function. These
corresponding functions must exist for each expression node and are produced by
calling that expression's `Func` method. These are usually called by the
function engine during function creation and validation.
corresponding functions must exist for each expression node and are produced as
the vertices of this returned graph. This is built for the function engine.

#### Function engine creation and validation

Expand Down
13 changes: 13 additions & 0 deletions examples/lang/map-iterator0.mcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import "iter"

$fn = func($x) { # notable because concrete type is fn(t1) t2, where t1 != t2
len($x)
}

$in1 = ["a", "bb", "ccc", "dddd", "eeeee",]

$out1 = iter.map($in1, $fn)

$t1 = template("out1: {{ . }}", $out1)

test $t1 {}
28 changes: 28 additions & 0 deletions examples/lang/map-iterator1.mcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import "datetime"
import "iter"
import "math"

$now = datetime.now()

# alternate every four seconds
$mod0 = math.mod($now, 8) == 0
$mod1 = math.mod($now, 8) == 1
$mod2 = math.mod($now, 8) == 2
$mod3 = math.mod($now, 8) == 3
$mod = $mod0 || $mod1 || $mod2 || $mod3

$fn = func($x) { # notable because concrete type is fn(t1) t2, where t1 != t2
len($x)
}

$in1 = if $mod {
["a", "bb", "ccc", "dddd", "eeeee",]
} else {
["ffffff", "ggggggg", "hhhhhhhh", "iiiiiiiii", "jjjjjjjjjj",]
}

$out1 = iter.map($in1, $fn)

$t1 = template("out1: {{ . }}", $out1)

test $t1 {}
2 changes: 2 additions & 0 deletions gapi/gapi.go
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,13 @@ type GAPI interface {

// Next returns a stream of switch events. The engine will run Graph()
// to build a new graph after every Next event.
// TODO: add context for shutting down to the input and change Close to Cleanup
Next() chan Next

// Close shuts down the GAPI. It asks the GAPI to close, and must cause
// Next() to unblock even if is currently blocked and waiting to send a
// new event.
// TODO: change Close to Cleanup
Close() error
}

Expand Down
179 changes: 0 additions & 179 deletions lang/ast/scope_test.go

This file was deleted.

Loading