Error after upgrading to Ember 2.4

[workmanw]

This error only occurs after the app has been active and running for a while, giving the user a chance to navigate between several different routes. It's very hard to reproduce, it seems to "just happen" after a while. We've been able to reproduce it a few times using our production build (ember.min.js), but never using a debug build (ember.debug.js).

Here is the stack:

 "Cannot read property '_lookupFactory' of undefined"

TypeError: Cannot read property '_lookupFactory' of undefined
    at i (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:7:2712)
    at o (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:7:2833)
    at Object.a [as default] (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:7:2888)
    at Object.i [as subexpr] (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:6:4717)
    at a (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:15:16476)
    at i (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:15:16302)
    at n (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:15:16189)
    at Object.r [as acceptHash] (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:15:16075)
    at n (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:15:26102)
    at Object.a.inline (https://qa-integration.batterii.com/assets/vendor-69da94618271be1c4338db3f0e942865.js:15:26664)

That seems to point back to the lookup-helper. The few times I've been lucky enough to catch this at a breakpoint, I've observed that the minified owner parameter becomes undefined. The rest of the env looks correct. See:

I know that's very little to go on. Short of coming up with an easy way to reproduce this, is there any additional information about the stack that might helpful in debugging this?

[workmanw]

What's even more odd is that you can see in the last screenshot, there is var s = "helper:" + e; on line 8786 ... but somehow on the very next line s is undefined. 😕 It's almost as if the stack is getting borked ... or the chrome debugger is just getting it wrong.

[raido]

I have seen something like this too. Happens rarely and i do not know how to reproduce. Also i am not sure if it is my app's problem or something else.

[workmanw]

@raido is your stack the same as mine (IE the failure is on _lookupFactory)? Are you on Ember 2.4.1? Have you seen this issue on prior versions of ember?

We upgraded from Ember 2.2 => Ember 2.4 and started seeing this issue pretty heavily in our server logs within a few days.

[rlivsey]

👍 seen this myself and confused the heck out of me, from my bugsnag logs:

11779      if (validateLazyHelperName(name, owner, env.hooks.keywords)) {
11780        var helperName = 'helper:' + name;
11781        if (owner.hasRegistration(helperName, options)) {
11782          helper = owner._lookupFactory(helperName, options);
11783        }
11784      }
11785    }

As you say, how can owner be undefined on line 11782 if it's got past the owner.hasRegistration the line before?

Likewise only seen it on production when minified (above comes from sourcemaps).

Logs show we've only seen it on Chrome so far.

[raido]

@workmanw Yes, my error is also related too _lookupFactory and is see lookupHelper in the stacktrace.

Logs from production build, happened just now with v2.3.0

TypeError: Cannot read property '_lookupFactory' of undefined
    at o (vendor-6292d0672068025de3c6d57c1fb505d0.js:7)
    at Object.a [as default] (vendor-6292d0672068025de3c6d57c1fb505d0.js:7)
    at Object.r [as lookupHelper] (vendor-6292d0672068025de3c6d57c1fb505d0.js:6)
    at Object.D [as inline] (vendor-6292d0672068025de3c6d57c1fb505d0.js:16)
    at Object.i.inline (vendor-6292d0672068025de3c6d57c1fb505d0.js:16)
    at l.populateNodes (vendor-6292d0672068025de3c6d57c1fb505d0.js:16)
    at l.render (vendor-6292d0672068025de3c6d57c1fb505d0.js:16)
    at i (vendor-6292d0672068025de3c6d57c1fb505d0.js:16)
    at vendor-6292d0672068025de3c6d57c1fb505d0.js:16
    at s (vendor-6292d0672068025de3c6d57c1fb505d0.js:16)

[jcbvm]

I can confirm this issue on 2.4.2, seeing it sometimes in production, not seen in development yet. Can this be caused by the ember-inspector maybe?

For more info: never had this on 2.3.x and the stack is the same (_lookupFactory)

EDIT: I can confirm that this is not a ember-inspector issue, error occured while inspector was disabled.

[gpoitch]

Confirmed bug in 2.4.1 and 2.4.2. Happens only with minified js

[workmanw]

@jcbvm @gdub22 Have either of you been able to reproduce it consistently? I've tried and tried to find at least a consistent set of steps in our app with the hope of building an ember-twiddle, but I've had no luck.

This is completely anecdotal and probably a red herring, but it seems to happen while trying to render a helper which has an ancestor component rendered with the component helper ({{component componentName}}).

[rlivsey]

@workmanw Likewise not been able to reproduce consistently, we do use the {{component}} helper a lot throughout our app.

[gpoitch]

@workmanw not 100% consistently but I think I narrowed it down to a particular component which does happen to use the component helper in its own template.

[ef4]

Uglify transpiles the function above to:

function n(e,t,r,n){
  var i=r.helpers[e];
  if(!i){
    var o=r.owner;
    if (a(e,o,r.hooks.keywords)){
      var s="helper:"+e;
      o.hasRegistration(s,n) && (i=o._lookupFactory(s,n));
    }
  }
  return i;
}

Notice that both property accesses on owner have been compiled down to a single line. My guess is that the crash actually happens on the first one, and the sourcemap fidelity is not high enough to distinguish them correctly.

Edited to add: Ah, but the crash is definitely for a _lookupFactory property, so my speculation must be wrong. Curiouser and curiouser.

[workmanw]

@ef4 Maybe ... but the few times I've caught this exception in the debugger, o (in your snippet) is undefined, but r.owner is a valid owner. In fact, I'm able to do r.owner.hasRegistration(s,n) && (r.owner._lookupFactory(s,n)); and get the class. Granted this is after Chrome has broken on a caught exception ... so the debugger could also be in a misleading state.

[rlivsey]

@ef4 yep it's definitely a strange one.

Could V8 be optimising it away for some reason? Who knows most about the dark arts of V8, @stefanpenner maybe?

[wycats]

If the line code that is crashing is:

o.hasRegistration(s,n) && (i=o._lookupFactory(s,n));

then:

1. o.hasRegistration(s,n) has already returned a truthy value; this means
2. o is neither null or undefined; therefore
3. Cannot read property '_lookupFactory' of undefined is either wrong or a bug in v8

Am I missing something obvious?

[wycats]

For people who have reproduced this problem, what version of Chrome are you running? Have you reproduced it in Firefox, IE, or Safari?

[workmanw]

@wycats I don't see anything obvious you're missing. These are the same conclusions I arrived at. We have only seen this issue logged on Chrome latest (48). In my reproduction testing, I was using 48.0.2564.116. I personally did not try Firefox, IE or Safari, but I will try them out and report back.

EDIT: I cannot simplify the problem to the point of producing a twiddle. But if it would be helpful, I can queue up one or more failures paused at a breakpoint, and jump on a screenhero if someone wants to poke around in the chrome debugger. Sometimes I can reproduce it 3 times in a minute. Sometimes it takes 20 minutes or more.

[workmanw]

@wycats Alright. I spent the last hour trying Chrome, Safari and Firefox. I was able to reproduce it several times in Chrome and not at all in Safari and Firefox. I'm not sure this is conclusive, but it is certainly pointing more and more toward a Chrome specific issue.

[wycats]

This whole thing reminds me of http://yehudakatz.com/2010/01/02/the-craziest-fing-bug-ive-ever-seen/

[raido]

I have not been able to reproduce it either. For me it typically happens when app boots. Reloading app several times probably crash it, yet it is not consistent. It may crash 5 times in just minutes or take half an hour. I tested on latest Chrome.

[stefanpenner]

@workmanw I would recommend forcing that function to stay in specific modes, this may help us find out more about the issue. At which point we should also ping our v8 friends. But for that, a reproduction (even a full app) is likely going to be required.

To keep functions in specific modes, you can slowly disable parts of v8, and see if the bug stops. This will help us get closer to the root cause.

first I would just disable inlining, by running chromes v8 with it disabled: --nouse_inlining (my guess is this may be sufficient)

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --js-flags="--nouse_inlining" --user-data-dir=/tmp/foobar

On the flip side, try disabling crankshaft all together --crankshaft=false

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --js-flags="--crankshaft=false" --user-data-dir=/tmp/foobar

[workmanw]

@stefanpenner

I tested this using Chrome 48 (rather than canary). If you'd like me to try with Canary, I'd be happy to.

With --nouse_inlining I was not able to reproduce this issue.
With --crankshaft=false I was able to reproduce this issue.

At which point we should also ping our v8 friends. But for that, a reproduction (even a full app) is likely going to be required.

I completely understand. I've spent about 4 hours trying to "work forwards" and build an Ember-twiddle that reproduces this. I just don't understand enough of what's going on to do that. So now I'm going to start "working backwards" using our app to simplify the reproduction by reducing the reproduction steps and ripping out as much out of our app as I can.

[stefanpenner]

With --nouse_inlining I was not able to reproduce this issue.

seems expected, so its most likely related to inlining bug of some kind.

With --crankshaft=false I was able to reproduce this issue.

this may be the wrong flag, i can't remember.

I completely understand. I've spent about 4 hours trying to "work forwards" and build an Ember-twiddle that reproduces this. I just don't understand enough of what's going on to do that. So now I'm going to start "working backwards" using our app to simplify the reproduction by reducing the reproduction steps and ripping out as much out of our app as I can.

@workmanw is it possible to share the app (or a URL to the app) as-is with steps to reproduce?
The reality is, reproducing this in isolation may be tricky.

[stefanpenner]

A :sadpanda: work-around, is to force the function to exceed the max AST that can be inlined. This should allow you to 🚢

putting the following string in that function's body, should do the trick (right now, these limits/heuristics change overtime)

"Pork chop porchetta rump, bacon turducken filet mignon tri-tip drumstick picanha beef ribs sausage salami. Leberkas beef landjaeger bresaola, sausage meatloaf pastrami frankfurter ribeye jowl turducken drumstick flank. Pork loin shank tongue leberkas ham strip steak salami swine short ribs cupim. Strip steak sausage turkey tenderloin, alcatra turducken porchetta ribeye brisket spare ribs rump salami ground round tail frankfurter. Kielbasa cow porchetta, hamburger jowl salami turducken capicola beef. Corned beef meatloaf ball tip landjaeger shank pork belly. Short loin kielbasa pig tail, brisket cupim salami andouille hamburger sausage short ribs."

[stefanpenner]

@workmanw if you can also check in canary, that would be handy.

[workmanw]

@stefanpenner

putting the following string in that function, should do the trick (right now, these can change overtime)

Your knowledge of the internals never cease to amaze me.

@workmanw is it possible to share the app as-is with steps to reproduce?

Yea, I should be able to make that happen. If you give me a little bit I'll provision some credentials to one of our pre-production environments and get the data primed. I could probably get okay to share the source if need be, but would have to do that privately.

@workmanw if you can also check in canary, that would be handy.

Done. I checked canary "version 51.0.2673.0 canary (64-bit)" without either of the flags and unfortunately I could still reproduce it.

[stefanpenner]

I could probably get okay to share the source if need be, but would have to do that privately.

It would help me personally try and reduce the problem further, although no guarantees. I more or less would love to do the following:

• prepare a reproduction for the V8 folks
• explore an interim [BUGFIX] solutions
• get a V8 report in (today/this weekend)

With a app in-front of me (where I can change code + explore) discovering a proper work-around/reproduction might be possible.

[workmanw]

EDIT: The application linked below is using a build of ember which contains the workaround from PR #13118. It is no longer be valid for reproduction of this issue. If anyone is interested in reproducing this issue using our app, contact me and I could probably make that happen.

---

@stefanpenner So I did whittle down the reproduction by injecting some code into the page. Believe me, otherwise it would have been a nightmare.

This is still not 100% reproducible. If you follow these steps and it still hasn't occurred, try restarting chrome all together.

1. Visiting this URL: https://qa-integration.batterii.com/#/community/MTpDb21tdW5pdHksOTAwMQ/room/MTpSb29tLDE5NzQ2MzAwMQ/wall/MTpSb29tLDE5NzQ2MzAwMSxXYWxsLDEwMDAx

2. Login with email: [email protected] and password: tomster1. That should deeplink you into a page that looks like this:

3. After logging in and landing on the above page, you'll need to refresh (so that you start clean from that page).

4. Open your chrome debugger and run the following in the console:

(function() {
var room = 'MTpSb29tLDE5NzQ2MzAwMQ',
    wall = 'MTpSb29tLDE5NzQ2MzAwMSxXYWxsLDEwMDAx',
    wallitem = 'MTpXYWxsSXRlbSwxOTg0NjMwMDQ';

function promiseTimer(ms) {
  return new Ember.RSVP.Promise(function(resolve) {
    Ember.run.later(resolve, ms);
  });
}

function timedTransition() {
  return BC.router.transitionTo.apply(BC.router, arguments).then(function() {
    return promiseTimer(800);
  });
}

function takeActions() {
  var downloadUrl = window.wallitemRecord.get('downloadUrl');
  window.open(downloadUrl);
  promiseTimer(1400).then(function() {
    return timedTransition('wall.wallitem', room, wall, wallitem);
  }).then(function() {
    return timedTransition('wall', room, wall);
  }).then(function() {
    return timedTransition('wall.wallitem', room, wall, wallitem);
  }).then(function() {
    return timedTransition('wall', room, wall);
  }).then(function() {
    return timedTransition('wall.wallitem', room, wall, wallitem);
  }).then(function() {
    return timedTransition('wall', room, wall);
  }).then(function() {
    return timedTransition('wall.wallitem', room, wall, wallitem);
  }).then(function() {
    return timedTransition('wall', room, wall);
  }).then(function() {
    return timedTransition('wall.wallitem', room, wall, wallitem);
  }).then(function() {
    return timedTransition('wall', room, wall);
  });
}

BC.store.findRecord('wallitem', wallitem).then(function(wallitem) { window.wallitemRecord = wallitem; });
$('<button id="crash-reproduce">Crash Reproduce</button>').appendTo('.top-right-nav');
$('#crash-reproduce').on('click', takeActions);
})();

5. Set Chrome Debugger to "Pause on Caught Exceptions".

6. The injected code should have added a button in the upper right corner. Click that button. First a new tab will open and trigger a file to be downloaded, then a modal dialog should open and close several times. That modal is actually "routable" so you should also observe the route changing. On the second or third open of the modal, you should hit the exception. If this doesn't occur, refresh the browser and try again (starting with step 4).

---

I'll work on getting you source code in the meantime. We run on Google App Engine so you will still have to connect to a cloud server, but I should be able to arrange it so you can run the client app locally (proxying to a cloud server).

Lastly, I'm on slack now and will be until about 4PM EST. I'd be happy to screenhero if need be. I'll also be available all day tomorrow.

[workmanw]

🎊 🎉 I tested @raido's PR and I cannot reproduce the issue. That does seem to be a successful workaround. 😄

Edit: Tested with Chrome 49 and Chrome 51.

[rwjblue]

I have been trying to follow the reported scenarios closely, but I believe this code is in Ember 2.3 also. Does Ember 2.3 also suffer this issue?

[raido]

Yes, i can confirm that Ember v2.3 is affected by this too.

[rwjblue]

OK, pulled #13118 into beta, release, and release-2-3 branches. The release and beta channels should get new builds shortly (via Travis) please bang on it for a while so we can confirm it does indeed fix this...

[2468ben]

@workmanw @raido Thank you so much for spending so much time reproducing the bug, working with Chromium to find & fix this issue.
It's one of those Heisenbugs where, trying to be a responsible OSS user, I waited to file an issue for over a month because I couldn't make a stable gist/twiddle/bin/etc. It just defied so much logic that I figured it must be my fault. Next time I'll be more proactive and ping the Slack room for fellow victims.

[stefanpenner]

Good job all!

@2468ben i wonder if we should maybe have a hesienbug/maybe vmbug label?

[stefanpenner]

[fixed by #13118]

[workmanw]

:)

@rwjblue I've updated my bower.json now to use "ember": "components/ember#9c3e5820" and I'm going to send it through the QA cycle. I'll let you know if any issues popup.

[rwjblue]

Thank you very much!

[chrisdoornink]

We've also been experiencing this issue and this appears to have fixed it. I've been hammering our site against the ember#9c3e5820 build for over an hour now and haven't seen any issues.

[rwjblue]

v2.4.3 has been released with the work around added in #13118.

[courthead]

Oh man, the hours I spent trying to reproduce/fix this bug before finding this thread... ugh. Good work guys!

[Turbo87]

we're currently running an app on [email protected] and we're hitting this exact issue according to Sentry even though the code includes the workaround in #13118 😞

[workmanw]

@Turbo87 This issue was also fix in Chrome as well. So there should only be 1 or 2 versions of Chrome that could run into this.

Are you sure it's the same issue?

[Turbo87]

@workmanw yeah, pretty sure it's the same. apparently some of our users are still running old Chrome versions and in fact some derived browsers (Sogou Explorer, Opera, Chromium, Dragon) are showing similar behavior according to our Sentry logs

[workmanw]

:(. I feel your pain. Some of our customers are enterprises that lock users into specific versions of Chrome and never let them upgrade.

It's possible that this issue resurfaced in some way. I can say with 100% certainty that, at the time, this workaround did resolve the issue for us (v2.4.3).

[raido]

Same here.

[krisselden]

@Turbo87 do you have specific versions?

[Turbo87]

• Chrome 49.0.2623
• Opera 36.0.2130
• Chromium 48.0.2564
• Sogou Explorer 1.0
• Chrome 48.0.2564
• Chrome 50.0.2632

Chrome 49 is by far the most common one for some reason

[givanse]

@Turbo87 did you ever found a way around this problem?

[Turbo87]

@givanse we've upgraded to Ember 2.12 by now and don't seem to have this issue anymore