Jump to content

[TOPIC: topicViewTemplate]
[GLOBAL: userSmallPhoto]
Photo

snapshot as paint input?
Started by roboward Oct 19 2013 01:44 PM

39 replies to this topic
[TOPIC CONTROLS]
Page 1 of 2 1 2
This topic has been archived. This means that you cannot reply to this topic.
[/TOPIC CONTROLS]
[modOptionsDropdown]
[/modOptionsDropdown]
[reputationFilter]
[TOPIC: post.html]
#1

roboward

[GLOBAL: userInfoPane.html]
roboward
  • Enthusiast

  • 50 posts
  • Corona SDK

Is there a way now to use a snapshot or other screen element as a paint input?

 

 

Right now it looks like inputs can only be image files, is this correct?
 
          paint1={ type="image", filename="blob1c.png" },
          paint2={ type="image", filename="blob1n2.png" }
 
Or perhaps there's an undocumented...   paint1 = {type = "snapshot",  snapshot1} or something
along those lines?
 
***********
 
In the case I"m thinking of I'd like to explicitly composite snapshotA over snapshotB (subtract, add, whatever), snapshotC over snapshotD, then composite the resulting B and D over whatever happens to be on the screen below it. 
 
 
 


[TOPIC: post.html]
#2

walter

[GLOBAL: userInfoPane.html]
walter
  • Moderator

  • 726 posts
  • Alumni

We've contemplated something like that. 

 

The question is where does the snapshot live? Our thinking is that this would go offscreen (i.e. not render) once the snapshot is used as another object's fill paint.



[TOPIC: post.html]
#3

rakoonic2

[GLOBAL: userInfoPane.html]
rakoonic2
  • Contributor

  • 503 posts
  • Corona SDK

Walter, regarding snapshots - have you given any more thoughts to my suggestion that they stop being an actual image (of sorts), and instead become a factory resource a la imagesheet, so you can spawn sprites from them? I can't think of any other conceptual way of having multiple images sharing a single snapshot that doesn't end up with a system that is alien to corona's current way of doing things?

 

 

I wouldn't assume that snapshots will just be used for rendering to the screen - at least, not directly. Their greatest strengths are to use them as more than just a filterable container anyway :)

 

What I envisage is that a snapshot works as it does, but you can't apply it directly to the screen. You'd need to spawn images from it, which gives you the ability to clone them. Something like:

 

local sprite = display.newImageRect( snapshotAsSource, width, height ).

 

Or, and this would be even more awesome - make it so you can create image sheets from a snapshot - this would add a TON of useful abilities (for example I am stuck in a demo I am working on because I need to create 10 strips from a single snapshot - currently the only way to do it is to cycle through them 1 at a time per frame).

Creating spritesheets and animations from a snapshot might not seem very intuitive, but being able to spawn sprites based on only sections of a snapshot would greatly increase flexibility, and make the snapshot very useful for setting up graphics to be re-used throughout the project.



[TOPIC: post.html]
#4

walter

[GLOBAL: userInfoPane.html]
walter
  • Moderator

  • 726 posts
  • Alumni

Yea, I agree this is going to enable some really amazing stuff!

 

So what if snapshots could be just as they are today, but also as a source for textures (both images and image sheets).

 

Here's what I'm thinking:

 

-- Create snapshot
local snapshot = display.newSnapshot()

...

-- Enable snapshot as a source for textures. Move snapshot offscreen. 
local textureId = graphics.defineTexture( { source=snapshot } )

-- snapshot is source for single image
object.fill = { type="image", texture=textureId }

-- snapshot is source for image sheet
local sheet = graphics.newImageSheet( { texture=textureId } )
object.fill = { type="image", sheet=sheet, frame=1 }


[TOPIC: post.html]
#5

chrislott

[GLOBAL: userInfoPane.html]
chrislott
  • Enthusiast

  • 39 posts
  • Corona SDK

I would love this as well!  We could do dynamic bump mapping and all sorts of fun stuff.



[TOPIC: post.html]
#6

gtt

[GLOBAL: userInfoPane.html]
gtt
  • Contributor

  • 164 posts
  • Corona SDK

This is very exciting for us! In general I am super excited about gfx 2.0 and we're hard at work on our new game based on it. 

In the reddit spirit I would just say "shut up and take my money!" :) 



[TOPIC: post.html]
#7

chrislott

[GLOBAL: userInfoPane.html]
chrislott
  • Enthusiast

  • 39 posts
  • Corona SDK

[TOPIC: post.html]
#8

rakoonic2

[GLOBAL: userInfoPane.html]
rakoonic2
  • Contributor

  • 503 posts
  • Corona SDK

Last two posts = most eloquence ever seen in this forum - I am in total agreement. Walter, my only concern is that I'm not fully conversant on the behaviour of fills - as long as an image based upon a snapshot (however it is done, via fill or cloning or something) distorts properly when you do things like distort in 2.75d etc then I'm going to be a happy bunny :)

[TOPIC: post.html]
#9

uuela9971

[GLOBAL: userInfoPane.html]
uuela9971
  • Enthusiast

  • 65 posts
  • Corona SDK

we are converting snapshot images into png files via Flash and using as image files, we have had some successful results with scaling small simple objects. We have not looked into any bump mapping yet. Testing the idea of setting these up in sprite sheets for object textures. We are going to try building the sprite sheets using texture packer.



[TOPIC: post.html]
#10

pxtracer

[GLOBAL: userInfoPane.html]
pxtracer
  • Observer

  • 5 posts
  • Corona SDK

Hi, I was also trying to know if possible to convert a snapshot into a sprite. There is any update in this subject? a year later? 

 

Thanks.



[TOPIC: post.html]
#11

Matthew Pringle

[GLOBAL: userInfoPane.html]
Matthew Pringle
  • Contributor

  • 660 posts
  • Corona SDK

Any news on this?



[TOPIC: post.html]
#12

Rob Miracle

[GLOBAL: userInfoPane.html]
Rob Miracle
  • Moderator

  • 25,558 posts
  • Enterprise

What are you looking for specifically?

 

Rob



[TOPIC: post.html]
#13

Matthew Pringle

[GLOBAL: userInfoPane.html]
Matthew Pringle
  • Contributor

  • 660 posts
  • Corona SDK

What Walter said above.

 

I need to be able to create a snap shot and then use that snapshot as a fill or in a composite filter as the 2nd paint channel.

 

The only way to do that now I believe would be to save and load the image each frame which would kill the framerate.



[TOPIC: post.html]
#14

vlads

[GLOBAL: userInfoPane.html]
vlads
  • Contributor

  • 657 posts
  • Corona Staff

Hello, everyone!
So here is a new feature. Give it a spin, we would like to get your feedback on it.
Half a year ago we introduced texture apis. Building on that API here is modifiable texture. It resembles Snapshot object, but it is quite different, even if APIs may seem similar.
 
New feature is TextureResourceCanvas type. It is in-memory texture, you can render stuff into it, and than assign it to other objects. Note that you can assign it to several different objects.
It is a subject to manual texture management, so beware of memory leaks. If you don't use it anymore, you have to release it. After you release it, you're loosing pointer Lua handle to it, but Display Objects using it would not become invalid.
 
I will put some code here to explain how it works.
 
Creating with newTexture:

local canvasTexture = graphics.newTexture( {
     type = "canvas"
     , width = 128
     , height = 128
     , pixelWidth = 256
     , pixelHeight = 256
} )

Here width and height here is dimensions of canvas where you would draw your objects, basically dimensions of visible world in canvas.
Pixel width/height - size of underlying texture. If omitted, this values would be selected to correspond to pixel dimensions of display object with width/height. You can set it to like 32 for cool pixely effects (with proper scaling modes), or just some other number, for example to save memory etc.
 
After it is created you can assign it to the fill to other objects. Use "canvasTexture.filename" and "canvasTexture.baseDir" pretty much anywhere where those parameters are expected. For example, in composite paint or creating image rect:
local circle = display.newCircle( display.contentCenterX, display.contentHeight*3/4, w/2 )
circle.fill = {
    type="composite",
    paint1={ type="image", filename="corona.png" },
    paint2={ type="image", filename=canvasTexture.filename, baseDir=canvasTexture.baseDir } -- magic here!
}
circle.fill.effect = "composite.phoenix"
 
local rect = display.newImageRect(
    canvasTexture.filename,  -- "filename" property required
    canvasTexture.baseDir,   -- "baseDir" property required
    display.contentWidth,
    display.contentHeight
)

 
To draw to canvas, you would use method canvasTexture:draw(). This would put your display objects off the screen and into internal queue. You have to manually update the texture with canvasTexture:invalidate().

This call would schedule to render objects in internal queue before next frame, and after rendering them it would move them to cache.
You can also set custom background colour. This colour would fill the texture when it is cleared.

canvasTexture:draw( someCircle )
canvasTexture:draw( someRect )
canvasTexture:setBackground( 0,0,0,1 )
canvasTexture:invalidate()

Note that (0; 0) point is in the centre of the canvas.
 
Thing to keep in mind is that when app is put in background, Android will make all GPU textures invalid, so you would have to redraw your canvas resource. Read documentation on how to do it.
 
Reading documentation is good in any way. It describes more edge cases and methods:
 
newTexture documentation <-- read me before using
TextureResourceCanvas documentation <-- me too

 

Canvas texture resources has a lot of fun applications, but also some limitations. Like nested textures can work weirdly, or native objects would not work at all.

Also, there is no possibility to add setPixel/getPixel. Textures are stored on GPUs and do not provide direct buffer access.

 
Also, I rewrote snapshot paint example, to use canvas. It has fire in the middle to demonstrate manual updates. It is attached to this post. Also, some other random example with star in it and composite paint.

Attached Files



[TOPIC: post.html]
#15

Matthew Pringle

[GLOBAL: userInfoPane.html]
Matthew Pringle
  • Contributor

  • 660 posts
  • Corona SDK

Hi Vlads

 

Thanks for this. Great stuff, I will be testing it to destruction shortly.

 

Matt



[TOPIC: post.html]
#16

Matthew Pringle

[GLOBAL: userInfoPane.html]
Matthew Pringle
  • Contributor

  • 660 posts
  • Corona SDK

So far everything seems fine. The texture seems to be sized a little different to just using a normal image as I needed to alter some settings in the shaders to get everything to line up again.

 

I will dig deep into the code, see if anything jumps out as being different or if its just my code.

 

Anyway I'm processing 2 x 1024 canvases, then sending them to 2 different shaders, each a composite paint at the moment ( but that might not be needed now ).

 

After that I take a snapshot of the result for additional processing on screen.

 

So far, testing on device I'm not seeing any different as to when I'm just using 1024 sized images.

 

Works as far back as an iPod Touch 5, which has the chip below the A7, and I can get 60fps no problem using optimised settings in the shaders.

 

 

So, all in all, this is perfect for what I need. It really opens up Corona to a lot of new techniques.



[TOPIC: post.html]
#17

StarCrunch

[GLOBAL: userInfoPane.html]
StarCrunch
  • Contributor

  • 796 posts
  • Corona SDK

@vlads Seems to blow up with masks.  :( Sprites seem okay, in my limited test. (Mixing it into the second example wasn't the best way to test.  :P )

 

Anyhow, I've got a ton of (image-based) use cases, so I'll report if anything turns up.



[TOPIC: post.html]
#18

vlads

[GLOBAL: userInfoPane.html]
vlads
  • Contributor

  • 657 posts
  • Corona Staff

I should had mentioned. Yes, for technical reasons you can not use TextureResourceCanvas as a mask (at this point). But you can set mask to objects using it as a fill without a problem.



[TOPIC: post.html]
#19

StarCrunch

[GLOBAL: userInfoPane.html]
StarCrunch
  • Contributor

  • 796 posts
  • Corona SDK

Picking some low-hanging fruit from among my ideas, here's a thing to paint "time" into a texture:

-- UV map thing

--
-- Permission is hereby granted, free of charge, to any person obtaining
-- a copy of this software and associated documentation files (the
-- "Software"), to deal in the Software without restriction, including
-- without limitation the rights to use, copy, modify, merge, publish,
-- distribute, sublicense, and/or sell copies of the Software, and to
-- permit persons to whom the Software is furnished to do so, subject to
-- the following conditions:
--
-- The above copyright notice and this permission notice shall be
-- included in all copies or substantial portions of the Software.
--
-- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-- EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-- MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-- IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-- CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-- TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-- SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
--
-- [ MIT license: http://www.opensource.org/licenses/mit-license.php ]
--

local CW, CH = display.contentWidth, display.contentHeight

local tex = graphics.newTexture{ type = "canvas", width = CW, height = CH }

tex:setBackground(0, 0, 0, 1)
tex:invalidate()

local img = display.newImageRect(tex.filename, tex.baseDir, CW, CH)

img:translate( display.contentCenterX, display.contentCenterY )

local previousX, previousY
local threshold = 2
local thresholdSq = threshold * threshold
local time = 0

local function draw (x, y)
	if time < 256 then
		local o = display.newImage("brush.png", x, y)

		o:setFillColor(time / 256, 0, 0)

		tex:draw(o)
		tex:invalidate("cache")

		previousX, previousY, time = x, y, time + 1
	end
end


local function listener (event)
	local x, y = event.x - img.x, event.y - img.y

	if event.phase == "began" then
		draw(x, y)
	elseif event.phase == "moved" then
		local dx = x - previousX
		local dy = y - previousY
		local deltaSq = dx^2 + dy^2

		if deltaSq > thresholdSq then
			draw(x, y)
		end
	end
end

Runtime:addEventListener("touch", listener)

--
do
	local kernel = { category = "composite", group = "uv_map", name = "basic" }

	kernel.vertexData = {
		{
			index = 0, name = "t", min = 0, max = 1, default = 0
		}
	}

	kernel.fragment = [[
		P_COLOR vec4 FragmentKernel (P_UV vec2 uv)
		{
			P_COLOR float r = texture2D(CoronaSampler1, uv).r;

			if (CoronaVertexUserData.x >= r) return vec4(0.); // check time vs. uv

			return CoronaColorScale(texture2D(CoronaSampler0, uv));
		}
	]]

	graphics.defineEffect(kernel)
end

local start = display.newCircle(CW * .6, CH * .7, 55)

start:setFillColor(1, 0, 0)
start:addEventListener("touch", function(event)
	if event.phase == "ended" then
		local r = display.newRect(display.contentCenterX, display.contentCenterY, CW, CH)

		r.fill = {
			type = "composite",
			paint1 = { type = "image", filename = "Image1.jpg" }, -- your image here
			paint2 = { type = "image", filename = tex.filename, baseDir = tex.baseDir }
		}

		r.fill.effect = "composite.uv_map.basic"

		img.isVisible, event.target.isVisible = false, false

		transition.to(r.fill.effect, { t = 1, time = 9000 }) -- can get reverse effect going 1 -> 0, of course
	end

	return true
end)

It takes an image of your choice and the brush from the CanvasPaint example (probably should have just used a circle, since the brush has its own alpha and you get a peculiar tapering fade).

 

When you run it, drag around on the screen to paint in time values, which gradually increase from 0 to 1, going into the red channel. Once it hits 1 it stops painting.

 

Click the red circle when you're ready and the effect's time threshold will transition from 0 to 1, unveiling pixels with the right time values. You can adjust the "// check time vs. uv" line to change the behavior.

 

This is a pretty boring proof of concept. The idea can get a lot fancier, though.



[TOPIC: post.html]
#20

StarCrunch

[GLOBAL: userInfoPane.html]
StarCrunch
  • Contributor

  • 796 posts
  • Corona SDK

@vlads I'm attempting something quite a bit more ambitious this time, which involves storing vertices in a texture.

 

I've tried to use vertex texture fetch. GLSL sources suggest this should "Just Work" provided the textures were already bound.

 

So far I've done something like this, in the fragment kernel:

CoronaSampler0 += 8.; // Shader fails to compile, but sampler name shows up in log

to suss out the sampler name. Then, after fixing that, in the vertex kernel:

uniform sampler2D u_FillSampler0;

P_POSITION vec2 VertexKernel (P_POSITION vec2 pos)
{
   // ... stuff

   P_POSITION vec2 new_pos = texture2DLod(u_FillSampler0, uv, 0.0).rg;

   return new_pos; // what our work gave us
}

This kind of thing compiles and runs, but I seem to be consistently getting "black", suggesting the sampler is unbound.

 

Are there complications in Corona's surrounding boilerplate, or am I just missing something obvious here?

 

(This is on Windows and I do have a way to detect that vertex texture units are available beforehand, although it's pretty clumsy: basically, a dummy pixel with an auxiliary shader that branches on gl_MaxVertexTextureImageUnits, followed by display.colorSample() and such.)

 

Also, might legitimate VTF support be a possibility? I think this would bring a lot of power in situations similar to my own.



[TOPIC: post.html]
#21

vlads

[GLOBAL: userInfoPane.html]
vlads
  • Contributor

  • 657 posts
  • Corona Staff

I don't see where you can bound this sampler. We're bounding textures to CoronaSampler0 (also to CoronaSampler0 if you use composite paint). Try using CoronaSampler0 instead of u_FillSampler0, and don't declare it, it's already declared & bound.

EDIT: I see it, it is in our sample. Well... basically CoronaSampler# is a synonym to u_FillSampler#. Use that instead and don't declare it.



[TOPIC: post.html]
#22

StarCrunch

[GLOBAL: userInfoPane.html]
StarCrunch
  • Contributor

  • 796 posts
  • Corona SDK

Argh, I must retract what I wrote.  :)

 

This much smaller test

-- 
do
	local kernel = { category = "generator", group = "custom", name = "vtf_test" }

	kernel.vertex = [[
		uniform sampler2D u_FillSampler0;

		P_UV varying vec3 v_uv;

		P_POSITION vec2 VertexKernel (P_POSITION vec2 p)
		{
			v_uv = texture2DLod(u_FillSampler0, CoronaTexCoord, 0.).rgb;

			return p;
		}
	]]

	kernel.fragment = [[
		P_UV varying vec3 v_uv;

		P_COLOR vec4 FragmentKernel (P_UV vec2 _)
		{
			return vec4(v_uv, 1.);
		}
	]]

	graphics.defineEffect(kernel)
end

local r = display.newImage("Image1.jpg")

r.x, r.y = display.contentCenterX, display.contentCenterY

r.fill.effect = "generator.custom.vtf_test"

is giving me exactly what I would expect (a blend of the four corner colors) for a variety of images, so apparently it DOES work. Time to restore some earlier code!

 

(Actually, it doesn't even seem picky about the name, only that there's a sampler there... and their order, I suppose, with two? I'll dig into the spec, when I have a chance.)

 

Would another option to system.getInfo() that returns GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS be possible?



[TOPIC: post.html]
#23

Matthew Pringle

[GLOBAL: userInfoPane.html]
Matthew Pringle
  • Contributor

  • 660 posts
  • Corona SDK

@vlads thanks for the help. I managed to achieve what I wanted, and more. Ive just up a post about it here https://forums.coronalabs.com/topic/61473-mode-7-3d-at-60fps/



[TOPIC: post.html]
#24

StarCrunch

[GLOBAL: userInfoPane.html]
StarCrunch
  • Contributor

  • 796 posts
  • Corona SDK

@vlads Well, I've got a good proof of concept going for what I was after: mesh test

 

It seems to be working well on Windows desktop but on somebody else's OSX machine it's not.  :( (On the latest daily build, too.) His screenshots from OSX were consistent with the texture not being read. I haven't attempted on a device.

 

This basically tests two approaches. In the first, when vertex texture fetch is available, positions and uvs are stored in a texture. Otherwise, positions (which must change) are stored in vertex userdata and uvs plus offsets-to-uvs are stored in the texture. Both techniques are used to sculpt some boring stock triangles into arbitrary ones dynamically.

 

Actually, if anybody else reading this would care to give the linked code a spin on Windows and / or OSX desktop (or device, I guess, though I'm not sure it's ready) and confirm (or not) what I said above, it'd be most appreciated!



[TOPIC: post.html]
#25

Matthew Pringle

[GLOBAL: userInfoPane.html]
Matthew Pringle
  • Contributor

  • 660 posts
  • Corona SDK

Here you go, latest OSX. Untitled_3.png




[topic_controls]
Page 1 of 2 1 2
 
[/topic_controls]