Jump to content

[TOPIC: topicViewTemplate]
[GLOBAL: userSmallPhoto]
Photo

What's wrong with my save function? Randomly crashes on devices - HELP!
Started by zolnier Apr 15 2019 12:24 PM

- - - - -
20 replies to this topic
[TOPIC CONTROLS]
[/TOPIC CONTROLS]
[modOptionsDropdown]
[/modOptionsDropdown]
[reputationFilter]
[TOPIC: post.html]
#1

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

Hey friends!  I'm having a terrible time diagnosing why my game save function is either randomly crashing devices or causing 0-length save files.  FYI, my save game files are json-based and can get pretty big (6-10MB+), but I'm not sure that should cause an issue.  Here's what I've got going on and I would most definitely appreciate any ideas!

 

SaveGame function - checks for valid GameID and then makes a backup of primary save file in case something goes wrong, then saves the file. Also creates an info file that shows key info about the save without having to load the whole big thing (used to show list of saves in main menu):

function SaveGame()
	local tmpT, y
	local showSave="N"
	local lfs = require "lfs"
	GameData.SaveGroup.alpha=1
	timer.performWithDelay(10000, function() GameData.SaveGroup.alpha=0 end,1)
	if _Game.GameID~=nil and _Game.GameID>0 then

		print("SAVING GAME!")
	
		tmpT={Team=_Game.UserTeamID, Year=_Game.SeasonYear, W=_Game.Teams[_Game.UserTeamID].Wins, L=_Game.Teams[_Game.UserTeamID].Losses, CW=_Game.Teams[_Game.UserTeamID].ConfWins, CL=_Game.Teams[_Game.UserTeamID].ConfLosses, C=_Game.CoachName,R=_Game.Teams[_Game.UserTeamID].Rank, CY=_Game.CareerYears}
		
		---Copy existing table as backup
		local destDir = system.DocumentsDirectory  -- Location where the file is stored
		local size = lfs.attributes (system.pathForFile( "game".._Game.GameID..".json", destDir ), "size")
		if size~=nil and size>0 then
			os.remove( system.pathForFile( "game".._Game.GameID..".bak", destDir ) )
			local result, reason = os.rename(
				system.pathForFile( "game".._Game.GameID..".json", destDir ),
				system.pathForFile( "game".._Game.GameID..".bak", destDir )
			)
		else
			os.remove( system.pathForFile( "game".._Game.GameID..".json", destDir ) )
		end
		
		saveTable(tmpT, "game".._Game.GameID.."info.json")
		
		if saveTable(_Game, "game".._Game.GameID..".json")==true then
			print("SAVED SUCCESSFULLY")
		else
			print("SAVE FAILED 1")
			--TRY AGAIN
			if saveTable(_Game, "game".._Game.GameID..".json")~=true then
				print("SAVE FAILED 2")
				os.remove( system.pathForFile( "game".._Game.GameID..".json", destDir ) )
				os.rename(
					system.pathForFile( "game".._Game.GameID..".bak", destDir ),
					system.pathForFile( "game".._Game.GameID..".json", destDir )
				)
			end
		end
	end
end

Here's the saveTable function that I use in the function above, as well. Yes, It's listed in my code before that SaveGame function!

function saveTable(t, filename)
	local path = system.pathForFile( filename, system.DocumentsDirectory)
	local file = io.open(path, "w")
	if file then
		local contents = json.encode(t)
		file:write( contents )
		io.close( file )
		--GameData.SaveGroup.alpha=0
		return true
	else
		--GameData.SaveGroup.alpha=0
		return false
	end
end


[TOPIC: post.html]
#2

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

FYI, _Game is the table that holds all of my game data.



[TOPIC: post.html]
#3

Prairiewest

[GLOBAL: userInfoPane.html]
Prairiewest
  • Observer

  • 26 posts
  • Corona SDK

According to Lua docs, functions like os.remove and os.rename will tell you if they failed - you're just ignoring the errors (if any).

http://lua-users.org/wiki/OsLibraryTutorial

 

You may want to add some vars that capture those return values, and hopefully see where things are failing.



[TOPIC: post.html]
#4

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

So, I went ahead and changed my save function to iterate through saves to automatically create backups to avoid any os library calls that might be creating issues on certain devices.  Only a few people are telling me that their game is crashing during the save process, but I want to eliminate any crashing.  Here is my updated SaveGame function:

function SaveGame()
	if _Game.SaveID==nil then
		_Game.SaveID=0
	end
	if _Game.GameID~=nil and _Game.GameID>0 then
		if _Game.SaveID==0 or _Game.SaveID==3 then
			print("SAVING GAME!  SaveID=1")
			_Game.SaveID=1
		elseif _Game.SaveID==1 then
			print("SAVING GAME!  SaveID=2")
			_Game.SaveID=2
		else
			print("SAVING GAME!  SaveID=3")
			_Game.SaveID=3
		end
		tmpT={Team=_Game.UserTeamID, Year=_Game.SeasonYear, W=_Game.Teams[_Game.UserTeamID].Wins, L=_Game.Teams[_Game.UserTeamID].Losses, CW=_Game.Teams[_Game.UserTeamID].ConfWins, CL=_Game.Teams[_Game.UserTeamID].ConfLosses, C=_Game.CoachName,R=_Game.Teams[_Game.UserTeamID].Rank, CY=_Game.CareerYears, SaveID=_Game.SaveID}
		saveTable(tmpT, "game".._Game.GameID.."info.json")
		
		if saveTable(_Game, "game".._Game.GameID.._Game.SaveID..".json")==true then 
			print("Game".._Game.GameID.._Game.SaveID.." SAVED SUCCESSFULLY") 
			GameData.Changed="N"
			timer.performWithDelay(1000, function() GameData.SaveGroup.alpha=0 end,1)
		else
			timer.performWithDelay(2000, function() GameData.SaveGroup.alpha=0 end,1)
		end
		
	end
	
end

Is there an easy way to "validate" my json data perhaps before saving?  I've really tried to make this as simple as possible, but for some reason, some people are still having crashing issues while saving... Thoughts?



[TOPIC: post.html]
#5

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

Just for clarity on how that new save function works, if a user opens save Game #1, it would will create up to 3 save file iterations as the user saves each time - first game11.json, then game12.json and then game13.json and will then re-save game11.json the fourth time, etc...



[TOPIC: post.html]
#6

Prairiewest

[GLOBAL: userInfoPane.html]
Prairiewest
  • Observer

  • 26 posts
  • Corona SDK

So you're checking the return value on all os calls now and you still don't see where it's failing?  Or, are you even able to replicate this issue on your own test devices?

 

Also, when are you trying to save these games?  Is it at application exit?  If so, it may be taking too long and your process gets killed.



[TOPIC: post.html]
#7

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

As you can see by my updated Save function, I have removed all os calls and streamlined into just iterating the save process.  The save is triggered either automatically after each "week" in game or on-demand via a save button (depending on which option a user chooses in settings).  The game does not autosave upon exit, so that is not an issue. It does prompt a user to save if they have not saved upon exit, but that's not the issue. I do fear there is something getting corrupted in the json conversion, so curious if there are ways to check and validate json data/keys as it is being written out to the file...



[TOPIC: post.html]
#8

Prairiewest

[GLOBAL: userInfoPane.html]
Prairiewest
  • Observer

  • 26 posts
  • Corona SDK

OK I see it's just io.open and write now.  You didn't mention if you're able to replicate the problem on your own devices?  Or are you just relying on reports from testers?

 

I've read and written JSON files to and from Lua tables many times, but have never had an issue like what you're describing.  And your saveTable code is almost exactly what is in the docs at https://docs.coronalabs.com/tutorial/data/jsonSaveLoad/index.html other than discarding any possible error string when opening the file.

 

According to the intro message here http://dkolf.de/src/dkjson-lua.fsl/home it's possible to have Lua tables that can't be encoded into JSON, but your tmpT table looks OK to me.  Maybe you can use rPrint (or tPrint, whatever you have like https://gist.github.com/ripter/4270799 ) to print your table out to console before saving it, then you'll see what's about to be saved.  Maybe the CoachName contains something that is breaking the JSOn encoding?



[TOPIC: post.html]
#9

Rob Miracle

[GLOBAL: userInfoPane.html]
Rob Miracle
  • Moderator

  • 25,779 posts
  • Enterprise

You can use many different online JSON validators to make sure your JSON file is valid. This is of course not practical in a distributed app, but while developing it, or even if you can catch the crash, have it email you the JSON file and then you can check it for errors. I'm pretty sure that our JSON library doesn't like mixed index types, such as a numerical index and a key index at the same level.

 

But what is likely happening is something is nil when you create the table to save:

tmpT={
    Team=_Game.UserTeamID, 
    Year=_Game.SeasonYear, 
    W=_Game.Teams[_Game.UserTeamID].Wins, 
    L=_Game.Teams[_Game.UserTeamID].Losses, 
    CW=_Game.Teams[_Game.UserTeamID].ConfWins, 
    CL=_Game.Teams[_Game.UserTeamID].ConfLosses, 
    C=_Game.CoachName,
    R=_Game.Teams[_Game.UserTeamID].Rank, 
    CY=_Game.CareerYears
}

If _Game.UserTeamID is nil or a value that doesn't exist in Game.Teams[], you're going to get an error that will crash Corona if not caught. Maybe check to make sure it's valid before trying to save things.

 

Something like:

if _Game.Teams and _Game.Teams[_Game.UserTeamID] and _Game.Teams[_Game.UserTeamID].Wins then
    -- create the temp table and save
end

might help harden things.

 

Rob



[TOPIC: post.html]
#10

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

Thanks, Rob!  I always like your perspective!   I'll look at your suggestion, but I believe the problem lies in saving the larger _Game table and not the data that I create and save in tmpT.  Is there a function that you've seen that will let me iterate through my _Game table (and sub-tables) and validate the data for json?  I fear something is not being set right in that table (which is pretty large - 3,500+ entries, with numerous values within each entry/row) and when things go wrong, the data is not being saved and is generating a 0-length save file.  So weird...



[TOPIC: post.html]
#11

nick_sherman

[GLOBAL: userInfoPane.html]
nick_sherman
  • Corona Geek

  • 1,803 posts
  • Corona SDK

For that amount of data is it worth moving to SqlLite? I don't like using JSON when that many rows are involved, plus you'll more easily be able to see what is being written out.

[TOPIC: post.html]
#12

Yoger Games

[GLOBAL: userInfoPane.html]
Yoger Games
  • Contributor

  • 109 posts
  • Corona SDK

If you think that the json encoding goes wrong, a starting point is to check if the encoding was successful (eg. not nil) and log an error if not successful.

function saveTable(t, filename)
...
    local contents = json.encode(t, {exception = function(...) 
        for i,v in ipairs(arg) do
            print("Exception: " .. tostring(v))
        end
    })

    if contents then
        file:write( contents )
        ...
    end

Edit: Didn't check if this works, small adjustments might be necessary.. :)


Edited by Yoger Games, 25 April 2019 - 09:21 AM.


[TOPIC: post.html]
#13

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

FYI, I used sqlLite in my first app and bulk updates were SOOOO slow.  Really made the app hang and created other issues.  I thought moving to in-memory and json might make it easier...



[TOPIC: post.html]
#14

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

Yoger, I like your suggestion but your code throws back errors...  "Unexpected symbol near }"



[TOPIC: post.html]
#15

nick_sherman

[GLOBAL: userInfoPane.html]
nick_sherman
  • Corona Geek

  • 1,803 posts
  • Corona SDK

Shouldn't be slow at all if you're wrapping your transactions in 'begin transaction' and 'commit'.

[TOPIC: post.html]
#16

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

nick, maybe it was because I was using a file-based database throughout.  Would it be better to load the entire database into memory and only write it back out to a file-based database from memory all at once during the "save" process if I decided to use sqllite again.   

 

Yoger, still want to get your exception check working... Any tweaks to fix that error?



[TOPIC: post.html]
#17

nick_sherman

[GLOBAL: userInfoPane.html]
nick_sherman
  • Corona Geek

  • 1,803 posts
  • Corona SDK

I haven't used in-memory db in Corona, just file-based. Multiple tables with 100,000+ records.

 

The slowest part of writing to SQLlite using lua is the string concatenation to build the query, so I had to optimise that quite a lot, building queries from strings in a table, using table.concat to build the final string.

 

The actual disk operations are pretty fast, especially on mobile. Windows desktop is 2-3 times slower.

 

Looks like you're working on a sports sim too so the use case is similar, I don't store much in memory - just grabbing and saving data when required, so I don't have a save game function.



[TOPIC: post.html]
#18

Yoger Games

[GLOBAL: userInfoPane.html]
Yoger Games
  • Contributor

  • 109 posts
  • Corona SDK

nick, maybe it was because I was using a file-based database throughout.  Would it be better to load the entire database into memory and only write it back out to a file-based database from memory all at once during the "save" process if I decided to use sqllite again.   

 

Yoger, still want to get your exception check working... Any tweaks to fix that error?

 

Yes, the function end statement was missing. 

 

Refactored slightly, now the code looks like:

local function exceptionHandling(...)
    print("json encode failed")
    for i,v in ipairs(arg) do
        print("Exception: " .. tostring(v))
    end
end

local contents = json.encode(t, {exception = exceptionHandling})

if contents then
    file:write( contents )
end

To test the function you can call 

exceptionHandling("test", "test2", "test3")

I tried but couldn't produce a lua table that triggered an exception in json encode.



[TOPIC: post.html]
#19

StarCrunch

[GLOBAL: userInfoPane.html]
StarCrunch
  • Contributor

  • 805 posts
  • Corona SDK

If your save files are 6-10 MB+, then it might be that between the "contents" variable and all the intermediate data (the JSON library's table, unique strings along the way, etc.) plus game state, you're just hitting a memory limit.

 

By the looks of it, encode will just table.concat() the final result, like what Nick described, at the very end. But apparently you can also pass in your own table through the options, via the buffer field, with bufferlen set to the size afterward. (This would distinguish the case where fewer than #buffer were written, if you recycled the table.) If memory is failing but only because it needs one big chunk, you could try this approach, manually concatenate sub-ranges, then send them to file:write() one by one.

 

Also, from a cursory examination it doesn't seem like the library is doing much with the table contents along the way, so you might even be able to proxy it right to your file (opened beforehand, of course), maybe with some tuned buffering. In theory this would lower the memory load a bit, at the cost of some complexity.

 

That said, all of the above is just an educated guess.  :)



[TOPIC: post.html]
#20

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

Wow, StarCrunch!  That's some good stuff... Some of this is new to me. I've never played around with the buffer.  Any suggestions on a function that would better "chunk" my save data or handle that buffering better?



[TOPIC: post.html]
#21

zolnier

[GLOBAL: userInfoPane.html]
zolnier
  • Enthusiast

  • 77 posts
  • Corona SDK

Just out of curiosity, is there a coroutine or something I can write that will check if the SQL command has completed?  I guess when worried about long SQL updates, tracking the query update job would be handy.  How have you implemented large sql updates and letting the user know that that it's updating and then letting them know when it's done.  Often, these large updates seem to "lock up" the app until then are done...




[topic_controls]
[/topic_controls]