Possible issue with different encodes

[spinscale]

Hey,

I have the following in my pom.xml

...
                                <data>
                                    <src>${project.basedir}/src/deb/default/</src>
                                    <type>directory</type>
                                    <excludes>.DS_Store</excludes>
                                    <mapper>
                                        <type>perm</type>
                                        <prefix>/etc/default</prefix>
                                        <user>root</user>
                                        <group>root</group>
                                    </mapper>
                                </data>
...

There is only one file inside which is in unix encoding. However, when the packaging is done under windows, this file suddenly gets converted to windows line.separators, which then makes it uninstallable under debian linux, as the it cannot be sourced anymore and bash returns and error code which in turn stops the installation. Also this does not seem to happen with every windows installation (works on two other machines).

I have only taken a quick peek at the source, but I thought the specified files are simply copied and not changed in any way (or is this possible with the template functionality)? Anything else which could cause this behaviour?

Any hints are greatly appreciated.

[tcurdt]

Odd - I thought we have the line ending under control.
Hm - might be that was just about the control files.

Can you try if you can reproduce it with the jdeb examples?

Thanks for reporting.

[costin]

Hi, the test run with the latest jdeb on Win 8.1 x64 is available at
https://gist.github.com/costin/cb1398be897f445e0f49

cheers,

[tcurdt]

@ebourg @tmortagne any of you guys have a windows machine at hand?

[tmortagne]

I can restart on Windows and try that later today I think. Just need to build jdeb with Maven, right ?

(will have to setup Window for all that, it's basically just a video game player OS for me right now)

At least I have Win 8.1 x64 too.

[tcurdt]

From the gist it seems like it. Building the examples deb could also be a good idea. Thanks!

[tmortagne]

Will test several versions of Maven, could be related to that too.

[tmortagne]

I can confirm that I get https://gist.github.com/costin/cb1398be897f445e0f49 too (both with Maven 3.0.5 and 3.2.1). I don't have much time right but I can try to debug it on Windows this weekend.

[tcurdt]

@tmortagne that would be fantastic. if you could just pinpoint the problem that would be a start.

[tmortagne]

As soon as I get some time to setup Eclipse on Windows I'm sure a step by step debug should give a lot of informations.

[tmortagne]

Hmm https://gist.github.com/costin/cb1398be897f445e0f49 is not related to encoding or line ending, it expect "C:\Users\THOMA_~1\AppData\Local\Temp\DataProducerFilesTestCase.16835971457892943192txt" and end up with "/Users/THOMA_~1/AppData/Local/Temp/DataProducerFilesTestCase.16835971457892943192txt" as filename parameter in DataConsumer#onEachFile call.

Looks more like test bug to me as this parameter is supposed to be the path in the tar archive and not the File#getAbsolutePath() used by the test which cannot work in Windows.

[tcurdt]

True. @tmortagne, could you fix the test as a first step?

[tcurdt]

Hm. I don't see how where we could possibly be converting the line endings. @spinscale is using a DirectoryDataProducer which calls produceFile and produceDir. Following produceFile one can see the Producers.produceInputStreamWithEntry passes the inputstream to the consumer. The anonymous consumer in DataBuilder is then just copying the stream.

So the possibly unrelated failing testcase aside I think we will need a way to reproduce this.

[tcurdt]

The example project has a similar data producer.
We should check to build the example on windows and see if it exhibits the same problems.

[tcurdt]

Too bad travis-ci does not support Windows yet. (IIUC)
travis-ci/travis-ci#216

[tcurdt]

Would be great if we could sneak this fix in before the next release (hopefully next week). But given we haven't really reproduced it yet I will not hold the release for it.

[tcurdt]

@spinscale (or @tmortagne or @costin) could you guys please try to build the example project on windows and check the line endings there?

[tmortagne]

Test fixed, hope I will get some time today to test the example.

[tmortagne]

I built https://github.com/tcurdt/jdeb/tree/master/src/examples/maven but not sure what I should look at. Can't find any txt file.

[tmortagne]

I have added a file with Unix line ending and when I build nothing is converted so I cannot reproduce the issue.

[tmortagne]

Ha wait, looks like 7zip does not see DEBIAN folder. Not sure which soft to use to see those files properly.

[tcurdt]

@tmortagne I would just use ar to uncompress the deb and then check the raw files.

[tmortagne]

It's ok I found http://www.altap.cz/salamander/features/deb-viewer-browser-extractor-windows/. So I can confirm that even the Debian files (which have Windows line ending in the sources in my case) have Unix line ending in the deb package so here too all seems ok.

[tcurdt]

Awesome. Thanks for investigating this. Then let's close this as "cannot reproduce". We can re-open this if we get a testcase ...and I will just ignore this for the upcoming release.

[ebourg]

@tmortagne You can use 7zip 9.10a to see the control files of a package. This feature was unfortunately removed in the recent releases :(

http://sourceforge.net/p/sevenzip/bugs/1130/

[tmortagne]

@ebourg OK good to know, very weird

[tombrown52]

I have an issue that sounds like this one. My entire repository is checked out from git with windows line endings. When jdeb builds the control.tar it correctly converts all control scripts from windows to unix line endings. But any scripts packaged in data.tar retain the windows line endings (no detection or conversion is done, the resources are opened simply with FileInputStream; https://github.com/tcurdt/jdeb/blob/master/src/main/java/org/vafer/jdeb/producers/AbstractDataProducer.java#L87). How can I package scripts that are not control scripts?

[tcurdt]

@tombrown52 at this jdeb does not support file conversion like that.

Either use git for this or (if the files are not changing that often) just tar them up and use those as input.

Feel free to add it as feature request for version 2 though #195

[tombrown52]

For others who stumble on this, I have solved it by running dos2unix on my data sh files via the postinst control script. I also declare dos2unix as a dependency to ensure it's present. A little ugly, but it works no matter how the line endings are or which system builds the .deb.

[tcurdt]

Really not the recommended way - but if it works for you... so be it.

[tombrown52]

I think in this case git is not the best tool to use for changing the line endings. They're text files; Having git treat them as binary, while functional, is technically incorrect. It also requires each contributor to correctly set line endings in their editor.

Is the tar file solution you suggest automated (e.g. using the assembly plugin), or are you suggesting manually creating a tar with .sh scripts?

[tcurdt]

@tombrown52 I recommend re-reading the article I linked. No one said they should be treated as binary. Using *.sh text eol=lf is indeed technical and functional correct.