ndarray to buffer converter fun

[jeromeetienne]

Long/complex explanation sorry: It explains why converting why generating protocol commands with vec3 always leads to exception.

---

This is the ndarray to buffer converter. It is used to convert vec3/vec4 to buffer, which is required to generate GSP commands from any visual.

GSP/gsp/matplotlib/__init__.py

Lines 43 to 51 in 6aefaab

	@register("ndarray", "Buffer")
	def ndarray_to_Buffer(obj):
	# WARN: Do we need to keep track of obj/Buffer such as not create
	# several buffers pointing at the same underlying object?
	# In the current implementation, Buffer is created each time
	# this convertes is called such that if the obj has been
	# mofidied, it shoudl be ok
	Z = glm.ndarray.tracked(obj)
	return Z._tracker.gsp_buffer

As it is a converter, the obj in there is a numpy.ndarray. It uses glm.ndarray.tracked(obj) which is supposed to be API compatible with np.ndarray (@rougier correct me if im wrong).

It then goes into the __new__ of https://github.com/vispy/GSP/blob/master/gsp/glm/ndarray/tracked.py.

GSP/gsp/glm/ndarray/tracked.py

Lines 36 to 40 in 3fe4d22

	def __new__(cls, *args, **kwargs):
	obj = np.ndarray.__new__(cls, *args, **kwargs)
	if cls.__tracker_class__ is not None:
	obj._tracker = cls.__tracker_class__(obj)
	return obj

And here notice how it call np.ndarray.__new__(cls, *args, **kwargs) which is relatively equivalent to np.ndarray(obj) with the obj from the converter.

Unfortunatly np.ndarray(other_ndarray) is not a valid API. here is the doc https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html which shows that the first argument is a shape . It interprets the other_ndarray as a definition of shape and lead to this error message

Exception has occurred: ValueError
maximum supported dimension for an ndarray is currently 64, found 200000

[rougier]

I confirm but I remember having taken this case into account with a priority somewhere explained somewhere in the numpy API. Need to remember where.

[rougier]

Ok, I was speaking about array_priority and array_wrap but this was for a different problme (in transform/measure.py)

[jeromeetienne]

@rougier If you could look on how to fix this bug, it would be great. here is a minimal python to reproduce.

right now it produces exceptions as describe above, the goal would be to generate a valid command file

import gsp
from gsp import core, visual, glm
import numpy as np

# Build a scene
canvas = core.Canvas(512, 512, 100.0)
viewport = core.Viewport(canvas, 0, 0, 512, 512, [1,1,1,1])
n = 200_000
P = glm.to_vec3(np.random.uniform(-1, +1, (n,2)), dtype=np.float32)
pixels = visual.Pixels(positions=P, colors=[0,0,0,1])
pixels.render(viewport)

# Save a command file
gsp.save(filename=f"pixels-2d.command.json",)

[rougier]

Can you try

    # Z = glm.ndarray.tracked(obj)
    Z = obj.view(glm.ndarray.tracked)

in the converter?

[jeromeetienne]

• @rougier i created a PR with your suggestion and a python script to test the generation/parsing of command files Trying to fix ndarray_to_buffer - DO NOT MERGE #16

It will helps tracking the dev

[rougier]

With your other PR concerning arg parse, do we really need this one ?
For any example, we shoudl be able to write "python example.py -json" > "read-and-exec.py -matplotlib" (or something like this). The read-and-exec would be a unique script for all examples since it does not depend on them.

[jeromeetienne]

• @rougier the PR is just a trick because i can not create branch on this repo, Trying to fix ndarray_to_buffer - DO NOT MERGE #16 should never be merged
• i updated the title of Trying to fix ndarray_to_buffer - DO NOT MERGE #16 to reflect that.

[rossant]

@rougier I can give @jeromeetienne the permission to create branches, ok?

[jeromeetienne]

For any example, we shoudl be able to write "python example.py -json" > "read-and-exec.py -matplotlib" (or something like this). The read-and-exec would be a unique script for all examples since it does not depend on them.

@rougier i dont understand what you mean, can you elaborate ?

[rougier]

With proper switch in args (provided we parse them), we could run any example to out a list of json commands. These command can be immediately redirected to a special script that read them and execute them. IF we specify a matplotlib backedn, this means we get matplotlib window with the rendered example. What I meant is that we need a unique reader for all examples.

[rougier]

@rossant No need to give right to create branch yet I think.

[jeromeetienne]

With proper switch in args (provided we parse them), we could run any example to out a list of json commands. These command can be immediately redirected to a special script that read them and execute them. IF we specify a matplotlib backedn, this means we get matplotlib window with the rendered example. What I meant is that we need a unique reader for all examples.

sorry im still lost, how this related to ndarray_to_buffer bug ?