Discussion:
Data transfer and picture file questions Dec. 16, 2009
(too old to reply)
E.D.G.
2009-12-16 22:07:59 UTC
Permalink
Posted by E.D.G. December 16, 2009

1. A Question Regarding Sending Data To Gnuplot
2. A Question Regarding Picture File Errors .gif .jpg etc.
3. A Computer Language For Scientists

1. A QUESTION REGARDING SENDING DATA TO GNUPLOT

A Perl program is being used in a Windows system to generate chart
data. It stores the plot data in text files on the computer hard drive.
Then it uses a "Pipe" to tell Gnuplot to plot specific data files.

This arrangement works well. However, one of the goals with the next
generation of the Perl program is to have it send Gnuplot data without
having to use hard drive files. The reason for wanting to do that is so
that people can be sent demonstration copies of the programs on CDs. And
when they run they will not create any data files on the computer hard
drive.

Data can be sent to Gnuplot through a pipe by using plot "-" type
commands. But while that works for sending commands, the transfer process
appears to be slow and prone to errors when large amounts of data are
transferred using a pipe.

Is there a better way to do this?

Can multiple data arrays be created within Gnuplot so that the
original data need be sent to Gnuplot only once. Then Gnuplot can be told
to plot a specific block of data in one of those arrays?

A "Ram Drive" could probably be created within the Windows memory and
files could be stored there instead of using the hard drive for that. But
that looks like it could get a little complicated.


2. A QUESTION REGARDING PICTURE FILE ERRORS .GIF .JPG ETC.

Charts are being plotted on the computer screen. When the data look
acceptable, Gnuplot is being told to create a jpg picture file with those
data.

A problem I have been having is that when Gnuplot is told to create a
jpg file that is larger than a certain size, it loses its resolution and
also contains errors. So, I am having to create smaller files and then
combine them using Windows Paint etc. gif files are having the same error.

That problem can be duplicated by creating a file for Gnuplot to plot
and store in a jpg file. Then keep copying and pasting the same data in the
original file again and again and running the program until the final jpg
picture finally contains errors.

Is there a limit to the size of a jpg file that Gnuplot can create?
Files that need to be created are presently in the 1000 x 2000 pixel range.
What I am doing at the moment is creating 1000 x 600 pixel jpg files and
then combining them using Windows Paint to get them to the right size.

3. A COMPUTER LANGUAGE FOR SCIENTISTS

Scientists, especially students doing scientific research, need a
free download computer language that they can easily use for research
applications not involving highly important projects such as forecasting the
weather etc. Perl and Gnuplot can be used together quite effectively for
such a language. But there are limitations, with easily understood
documentation for Perl being perhaps the main one. FreeBasic works quite
well. The documentation is easily understood. But it is not being
supported by any formal organization as far as I can tell. And it has other
limitations.

At the moment I myself do not have time to work on this, but it might
be worthwhile for someone to ask the ActiveState company that makes free
Perl download versions available if they would be willing to create a
downloadable version that contains all of the resources that scientists
would like. Once you learn how to get through the documentation and write
Perl programs, it can be quite versatile, powerful, and easy to use. I
myself have never made a major effort to learn how the Perl graphics work
because I could not understand the documentation. So I decided to use
Gnuplot for graphics.

These are personal opinions.
Witold Baryluk
2009-12-18 01:47:07 UTC
Permalink
Posted by E.D.G.   December 16, 2009
1.  A Question Regarding Sending Data To Gnuplot
2.  A Question Regarding Picture File Errors .gif .jpg  etc.
3.  A Computer Language For Scientists
1.  A QUESTION REGARDING SENDING DATA TO GNUPLOT
       A Perl program is being used in a Windows system to generate chart
data.  It stores the plot data in text files on the computer hard drive.
Then it uses a "Pipe" to tell Gnuplot to plot specific data files.
       This arrangement works well.  However, one of the goals with the next
generation of the Perl program is to have it send Gnuplot data without
having to use hard drive files.  The reason for wanting to do that is so
that people can be sent demonstration copies of the programs on CDs.  And
when they run they will not create any data files on the computer hard
drive.
       Data can be sent to Gnuplot through a pipe by using plot "-" type
commands.  But while that works for sending commands, the transfer process
appears to be slow and prone to errors when large amounts of data are
transferred using a pipe.
Is there a better way to do this?
       Can multiple data arrays be created within Gnuplot so that the
original data need be sent to Gnuplot only once.  Then Gnuplot can be told
to plot a specific block of data in one of those arrays?
       A "Ram Drive" could probably be created within the Windows memory and
files could be stored there instead of using the hard drive for that.  But
that looks like it could get a little complicated.
Hi,

this question was going to be my 2nd proposeal for gnuplot.

One of the possibilities is to use tmpfs on Linux, it is essentially
"RamDrive".

The problem with plot "-" is that it is quite slow (even with
buffering), and many things
doesn't work easly, like replot, zooming, scalling in wx/x11.

My proposal was to have a way to create shared memory region, shared
beetwen gnuplot and other application(s), and tell gnuplot to treat it
just like
file which was already read into the buffer. One can use this way also
binary formats,
making very compact representation.


Is it possible?

I was thinking abou some hacks like some files in /proc/self/ but
can't find anything
usefull now.

Such approach can be easly hacked and used, but have few
disadvantages:
- gnuplot will read it to the own datastructures, so data essentially
will be represented
twice in memory.


Ethan's proposal about streaming can make this problem dissapear
(iterate over it directly and send to the proper procedured), it will
also allow something even better:
- use shared memory only for streaming buffers, ie.
1. start application A
2. create nacassary shared memories
3. create pipes
4. fork
5. start gnuplot and do not close shmem
6. send setup of shmem to gnuplot
7. send "plot 'shmem:1' streaming" command
8. fill shmem buffer,
9. send "filled shmem:1"
10. gnuplot reades it, sends partial data to the terminal
(drawing), and gives back prompt ">"
11. send "filled shmem:1"
12. gnuplot reades it again, sends next partial data to the
terminal (drawing), and gives back prompt ">"
...
13. send "clear shmem:1"

This can allow drawing enourmous ammount of points, without ussing
excesive memory, and do it fast.

Just my proposition.
Witold Baryluk
2009-12-18 02:25:53 UTC
Permalink
Post by Witold Baryluk
       A "Ram Drive" could probably be created within the Windows memory and
files could be stored there instead of using the hard drive for that.  But
that looks like it could get a little complicated.
Hi,
this question was going to be my 2nd proposeal for gnuplot.
One of the possibilities is to use tmpfs on Linux, it is essentially
"RamDrive".
The problem with plot "-" is that it is quite slow (even with
buffering), and many things
doesn't work easly, like replot, zooming, scalling in wx/x11.
My proposal was to have a way to create shared memory region, shared
beetwen gnuplot and other application(s), and tell gnuplot to treat it
just like
file which was already read into the buffer. One can use this way also
binary formats,
making very compact representation.
Is it possible?
I was thinking abou some hacks like some files in /proc/self/ but
can't find anything
usefull now.
Such approach can be easly hacked and used, but have few
 - gnuplot will read it to the own datastructures, so data essentially
will be represented
   twice in memory.
Ethan's proposal about streaming can make this problem dissapear
(iterate over it directly and send to the proper procedured), it will
 - use shared memory only for streaming buffers, ie.
     1. start application A
     2. create nacassary shared memories
     3. create pipes
     4. fork
     5. start gnuplot and do not close shmem
     6. send setup of shmem to gnuplot
I found possible way to to this:

2. int fd = shm_open("/some_uniq_name_1", O_CREAT | O_RDWR, 0600);
2a. fcntl(fd, F_SETFD, 0); // don't close it on exit (after fork &
execve)
...
7. fprintf(gnuplot_pipe, "plot '/proc/self/fd/%s' with points", fd);
which is equivalent to
7. fprintf(gnuplot_pipe, "plot '/dev/shm/some_uniq_name_1' with
points");
if shm is monted like [ tmpfs on /dev/shm type tmpfs
(rw,nosuid,nodev) ]

Because in such situation /proc/self/fd/%s is symbolic link to the /
dev/shm/some_uniq_name_1.

Second option (using /dev/shm) is more convinient because we don't
need to execute step 2a.

Works properly without even recompiling gnuplot!

ftruncate can be needed in case a file is smaller after overwrite.
shm_unlink for removeing file.
mmap and msync can be usefull in writing side so it is not needed to
use buffering or writing using "write/fprintf" functions.

Still there is change that memory isn't synchronized properly
(especially on multicore systems), but it is probably minor problem.

Regards,
Witek
E.D.G.
2009-12-18 21:12:39 UTC
Permalink
Post by Witold Baryluk
My proposal was to have a way to create shared memory region, shared
Hi Witold,

Thanks for the recommendations.

Someone posting a response to my Ram Drive question in the Windows XP
Newsgroup also recommended using shared memory. And he provided the
following reference:

http://msdn.microsoft.com/en-us/library/aa366551%28VS.85%29.aspx

It sounds promising. But it is something I will have to gradually do
some work on as I am not familiar with the programming.

My other question pertained to the size of the jpg picture files that
Gnuplot can create. I am wondering if there is a picture file size
limitation built into Gnuplot or if I might be using the wrong code.
sfeam
2009-12-18 22:32:33 UTC
Permalink
Post by E.D.G.
My other question pertained to the size of the jpg picture files that
Gnuplot can create. I am wondering if there is a picture file size
limitation built into Gnuplot or if I might be using the wrong code.
First point: Gnuplot creates JPEG images by calling into the external
library libgd. There is no limit on the size of the image that can
be created short of system-dependent limits on file size (e.g. 4GB on
a FAT16 filesystem).

Second point: JPEG is in general not well suited to graphs. It is an
intrinsically lossy format that exhibits bad aliasing effects on
thin lines. Graphs are full of these. Use PNG instead.
E.D.G.
2009-12-19 17:28:40 UTC
Permalink
"sfeam" <***@users.sourceforge.net> wrote in message news:hggvu5$3qn$***@news.eternal-september.org...

Posted by E.D.G. December 19, 2009
Post by sfeam
thin lines. Graphs are full of these. Use PNG instead.
Thanks for the information.

Giving PNG a try sounds easy enough.
E.D.G.
2009-12-29 10:31:01 UTC
Permalink
Post by sfeam
Second point: JPEG is in general not well suited to graphs. It is an
intrinsically lossy format that exhibits bad aliasing effects on
thin lines. Graphs are full of these. Use PNG instead.
Posted by E.D.G. December 29, 2009

Making the move from jpg to png format files did not solve the
resolution problem I have been having with large picture files. However,
the png files do appear to have a much better appearance. jpg and gif files
had something like a shadow or ghost image with both the data lines and
text. So I am planning to make a permanent move to using png files.

Regarding the resolution problem:

Since moving to png did not solve it the next step was to take a more
detailed look at the data that my Perl program was generating. And that
solved the problem.

The early versions of the Perl program were designed to create
relatively small picture files. At the time those programs were created I
wasn't sure if Gnuplot could work with decimal type numbers when it plotted
the data. And I was having a problem with plotting speed. So the Perl
program was directed to send Gnuplot only integer numbers.

When the attempt was made to create those much larger files, the
plotting speed problem had already been solved. But what I have now
discovered is that the integer numbers are not sufficiently accurate for
those especially large charts. So, the Perl program has been modified so
that decimal type numbers are being sent to Gnuplot. It appears to be
running okay with them. The resolution problem has been solved. And there
are no new plotting speed problems.

Continue reading on narkive:
Loading...