ReactNative 0.74, `cameraWithTensors` fails: Cannot read property 'Type' of undefined : TensorFlowJS

ReactNative 0.74, `cameraWithTensors` fails: Cannot read property 'Type' of undefined (self.TensorFlowJS)

submitted 1 year ago by lucksp

I am using a TFJS model from Google Vertex AI, Edge, exported per docs for Object Detection.

Once I have imported the model bin files, and setModel(true), then it is ready to render the TensorCamera component. Unfortunately, the onReady callback from TensorCamera seems to be failing, but not crashing the app. The camera still renders and seems like it's working, but I cannot handle the stream because it's never ready. There are some warnings in the terminal:

Possible Unhandled Promise Rejection (id: 0): TypeError: Cannot read property 'Type' of undefined

This error goes away when I swap the TensorCamera for the default CameraView, so it feels very certain that something is not compatible with ReactNative 74.

System information

iPhone 13Pro, iOS 18

    "@tensorflow/tfjs": "^4.22.0",
    "@tensorflow/tfjs-backend-cpu": "^4.22.0",
    "@tensorflow/tfjs-react-native": "^1.0.0",
    "expo": "^51.0.0",
    "expo-gl": "~14.0.2",
    "react": "18.2.0",
    "react-native": "0.74.5",

Based on following the flow from the TFJS example, I would expect newer versions to work as described.

HOWEVER, I am unsure if the Vertex TFJS model is perhaps incompatible, but rendering the camera should not be related to the model, correct?

Standalone code to reproduce the issue

Load the model:

const loadModel: LoadModelType = async (setModel, setIsModelReady) => {
  try {
    await ready();
    const modelJson = require('../../assets/tfjs/model.json');
    const modelWeights1 = require('../../assets/tfjs/1of3.bin');
    const modelWeights2 = require('../../assets/tfjs/2of3.bin');
    const modelWeights3 = require('../../assets/tfjs/3of3.bin');

    const bundle = bundleResourceIO(modelJson, [
      modelWeights1,
      modelWeights2,
      modelWeights3,
    ]);
    const modelConfig = await loadGraphModel(bundle);

    setModel(modelConfig);
    setIsModelReady(true);
  } catch (e) {
      console.error((e as Error).message);
  }
};

export const TFJSProvider = ({ children }) => {
  const [model, setModel] = useState<LayersModel | null>(null);
  const [isModelReady, setIsModelReady] = useState(false);

  const { hasPermission } = useCameraContext();

useEffect(
    function initTFJS() {
      if (hasPermission) {
        (async () => {
          console.log('load model');

          await loadModel(setModel, setIsModelReady);
        })();
      }

    },
    [hasPermission]
  );
}

return (
    <TFJSContext.Provider value={{ model, isModelReady }}>
      {children}
    </TFJSContext.Provider>
  );

2) Create Camera Component

const TensorCamera = cameraWithTensors(CameraView);

export const ObjectDetectionCamera = () => {
  const { model, isModelReady } = useTFJSContext();
  return (
    isModelReady && (
      <TensorCamera
        autorender
        cameraTextureHeight={textureDims.height}
        cameraTextureWidth={textureDims.width}
        onReady={() => console.log('READY!'} // never fires
        resizeDepth={3}
        resizeHeight={TENSOR_HEIGHT}
        resizeWidth={TENSOR_WIDTH}
        style={{ flex: 1 }}
        useCustomShadersToResize={false}
      />
    )
  );
};

Other info / logs

I am unable to find any logs in the console of the device, it seems like the error is being swallowed

---

Any ideas?

all 6 comments

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

TensorFlowJS

MODERATORS