Prevent freezing while working with large files?

Questions about the LÖVE API, installing LÖVE and other support related questions go here.
Forum rules
Before you make a thread asking for help, read this.
KayleMaster
Party member
Posts: 223
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster »

You can still do that in C, using FFI, if you absolutely need to. I don't think loading a 600k line file into a contiguous array would work well tho.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

Well, at least in VB.net I do kinda the same, using the same files to read.

Code: Select all

Dim DataFiles() As String = IO.Directory.GetFiles("data", "*.sav")
For iFiles = 0 To DataFiles.Length - 1
  Dim Lines() As String = IO.File.ReadAllLines(DataFiles(iFiles))
  For iLines = 0 To Lines.Length - 1
    Line = Lines(iLines)
  Next
Next
This iterates all files in 4 seconds, while the love attempt takes several minutes for 1MB+ files, even without any graphical extras.
KayleMaster
Party member
Posts: 223
Joined: Mon Aug 29, 2016 8:51 am

Re: Prevent freezing while working with large files?

Post by KayleMaster »

I mean, I can optimize it if you need to. But I'd need your project folder and data to do that.
Anyways, Love2D isn't really meant for this, obviously it's a gamedev framework, but I don't think it's that slow.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

Yeah, I know it's not meant for this. But I try to solve tasks I think are relatively simple with lua/love to get used to the framework, imgui design and lua in general.
A friend of mine is working on a machine learning engine and I will provide dev tools and ui frontend for a game using it later (or maybe someday :D), so I try to do as much in lua/love as possible, even if it's a bit out of the ordinary.

Sadly, I can't post the data files here, cause there's some licensing involved (they're from a game we're modding) but I could PM you if you're interested.
grump
Party member
Posts: 716
Joined: Sat Jul 22, 2017 7:43 pm

Re: Prevent freezing while working with large files?

Post by grump »

typx wrote: Wed Jun 20, 2018 9:48 am This iterates all files in 4 seconds, while the love attempt takes several minutes for 1MB+ files, even without any graphical extras.
You must be running this on a potato or you're doing something severely wrong.

Code: Select all

local start = love.timer.getTime()

local lines = {}
for i = 1, 600000 do lines[i] = ('x'):rep(100) end
love.filesystem.write('test.txt', table.concat(lines, '\n'))

lines = {}
for line in love.filesystem.lines('test.txt') do
	lines[#lines + 1] = line
end

print(#lines, "lines:", love.timer.getTime() - start, "seconds")
Output:

Code: Select all

600000	lines:	0.8547670400003	seconds
Generating a table of 600k lines with 100 chars each, joining them, writing them to a file, reading them back in and put them in a table takes less than one second for me. That's 57 MB of data. Granted, the data is still in the filesystem cache when reading it back.

Are you using LÖVE 11? 0.10.2 and below had a bug in love.filesystem.lines that made it perform extremely poorly when reading from a love file.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

Ah, yeah I'm using 0.10.2, meh :|
Will give it a try with 11.1 tomorrow.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

Updating love helped a lot, reading just takes several seconds now :D Sadly, when I try to manipulate and rewrite the files now, it's getting super slow again :| love.filesystem.append single lines and also love.filesystem.write a huge string variable are unusable. I know writing to a table and just saving as lua is the way to go, but since we're modding an existing game which expects textfiles, that's not an option.
User avatar
bartbes
Sex machine
Posts: 4946
Joined: Fri Aug 29, 2008 10:35 am
Location: The Netherlands
Contact:

Re: Prevent freezing while working with large files?

Post by bartbes »

Can't you just get an actual file handle in append mode and keep appending that way? Sure sounds more efficient than repeatedly opening and closing the same file.
typx
Prole
Posts: 20
Joined: Fri Apr 06, 2018 1:26 pm

Re: Prevent freezing while working with large files?

Post by typx »

Oh yeah, that did speed up everything a lot. Thanks to all of you :D
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 53 guests