Skip to content

Parsing compatibility of the FTP LIST command for Windows servers #439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
althonos merged 9 commits into from
Jan 8, 2021
Merged

Parsing compatibility of the FTP LIST command for Windows servers #439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
19e4387
Issue #438: Extended parsing compatibility of the FTP `LIST` command …
Nov 5, 2020
3bbbc48
Adding to the previous commit, performed minor changes according to s…
Nov 5, 2020
7b76fb1
Sorted CONTRIBUTORS.md alphabetically.
Nov 19, 2020
d1997ba
Cleaned up some redundant and unclear code.
Nov 19, 2020
89d0919
fs._ftp_parse, removed some code that was only assumed to be necessar…
Nov 20, 2020
c04ff0a
Re-merged two functions in fs._ftp_parse.
Nov 23, 2020
4a19bb8
Removed `_find_suitable_format` after its only use was deleted in the…
Nov 23, 2020
d39b985
Fixed a typo in a variable name in fs._ftp_parse by completely removi…
Dec 12, 2020
1fa1f27
Removed redundant line in fs._ftp_parse.
Dec 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
- Missing `mode` attribute to `_MemoryFile` objects returned by `MemoryFS.openbin`.
- Missing `readinto` method for `MemoryFS` and `FTPFS` file objects. Closes
[#380](https://github.com/PyFilesystem/pyfilesystem2/issues/380).
- Added compatibility if a Windows FTP server returns file information to the
`LIST` command with 24-hour times. Closes [#438](https://github.com/PyFilesystem/pyfilesystem2/issues/438).

### Changed

Expand Down
3 changes: 2 additions & 1 deletion CONTRIBUTORS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,15 @@

Many thanks to the following developers for contributing to this project:

- [Andreas Tollkötter](https://github.com/atollk)
- [C. W.](https://github.com/chfw)
- [Diego Argueta](https://github.com/dargueta)
- [Geoff Jukes](https://github.com/geoffjukes)
- [Giampaolo](https://github.com/gpcimino)
- [Justin Charlong](https://github.com/jcharlong)
- [Louis Sautier](https://github.com/sbraz)
- [Martin Larralde](https://github.com/althonos)
- [Morten Engelhardt Olsen](https://github.com/xoriath)
- [Nick Henderson](https://github.com/nwh)
- [Will McGugan](https://github.com/willmcgugan)
- [Zmej Serow](https://github.com/zmej-serow)
- [Morten Engelhardt Olsen](https://github.com/xoriath)
37 changes: 25 additions & 12 deletions fs/_ftp_parse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,17 @@
RE_WINDOWSNT = re.compile(
r"""
^
(?P<modified>.*?(AM|PM))
\s*
(?P<size>(<DIR>|\d*))
\s*
(?P<modified_date>\S+)
\s+
(?P<modified_time>\S+(AM|PM)?)
\s+
(?P<size>(<DIR>|\d+))
\s+
(?P<name>.*)
$
""",
re.VERBOSE)
re.VERBOSE,
)


def get_decoders():
Expand Down Expand Up @@ -82,15 +85,13 @@ def parse_line(line):


def _parse_time(t, formats):
t = " ".join(token.strip() for token in t.lower().split(" "))

_t = None
for frmt in formats:
try:
_t = time.strptime(t, frmt)
break
except ValueError:
continue
if not _t:
else:
Copy link
Contributor

@lurch lurch Dec 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oooh, I've just learned about the for ... else ... syntax in Python which I wasn't aware of before 🙂

return None

year = _t.tm_year if _t.tm_year != 1900 else time.localtime().tm_year
Expand All @@ -104,6 +105,10 @@ def _parse_time(t, formats):
return epoch_time


def _decode_linux_time(mtime):
return _parse_time(mtime, formats=["%b %d %Y", "%b %d %H:%M"])


def decode_linux(line, match):
perms, links, uid, gid, size, mtime, name = match.groups()
is_link = perms.startswith("l")
Expand All @@ -114,7 +119,7 @@ def decode_linux(line, match):
_link_name = _link_name.strip()
permissions = Permissions.parse(perms[1:])

mtime_epoch = _parse_time(mtime, formats=["%b %d %Y", "%b %d %H:%M"])
mtime_epoch = _decode_linux_time(mtime)

name = unicodedata.normalize("NFC", name)

Expand All @@ -138,12 +143,18 @@ def decode_linux(line, match):
return raw_info


def _decode_windowsnt_time(mtime):
return _parse_time(mtime, formats=["%d-%m-%y %I:%M%p", "%d-%m-%y %H:%M"])


def decode_windowsnt(line, match):
"""
Decodes a Windows NT FTP LIST line like these two:
Decodes a Windows NT FTP LIST line like one of these:

`11-02-18 02:12PM <DIR> images`
`11-02-18 03:33PM 9276 logo.gif`

Alternatively, the time (02:12PM) might also be present in 24-hour format (14:12).
"""
is_dir = match.group("size") == "<DIR>"

Expand All @@ -161,7 +172,9 @@ def decode_windowsnt(line, match):
if not is_dir:
raw_info["details"]["size"] = int(match.group("size"))

modified = _parse_time(match.group("modified"), formats=["%d-%m-%y %I:%M%p"])
modified = _decode_windowsnt_time(
match.group("modified_date") + " " + match.group("modified_time")
)
if modified is not None:
raw_info["details"]["modified"] = modified

Expand Down
66 changes: 48 additions & 18 deletions tests/test_ftp_parse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,18 @@ class TestFTPParse(unittest.TestCase):
@mock.patch("time.localtime")
def test_parse_time(self, mock_localtime):
self.assertEqual(
ftp_parse._parse_time("JUL 05 1974", formats=["%b %d %Y"]),
142214400.0)
ftp_parse._parse_time("JUL 05 1974", formats=["%b %d %Y"]), 142214400.0
)

mock_localtime.return_value = time2017
self.assertEqual(
ftp_parse._parse_time("JUL 05 02:00", formats=["%b %d %H:%M"]),
1499220000.0)
ftp_parse._parse_time("JUL 05 02:00", formats=["%b %d %H:%M"]), 1499220000.0
)

self.assertEqual(
ftp_parse._parse_time("05-07-17 02:00AM", formats=["%d-%m-%y %I:%M%p"]),
1499220000.0)
1499220000.0,
)

self.assertEqual(ftp_parse._parse_time("notadate", formats=["%b %d %Y"]), None)

Expand Down Expand Up @@ -164,39 +165,68 @@ def test_decode_linux(self, mock_localtime):
def test_decode_windowsnt(self, mock_localtime):
mock_localtime.return_value = time2017
directory = """\
unparsable line
11-02-17 02:00AM <DIR> docs
11-02-17 02:12PM <DIR> images
11-02-17 02:12PM <DIR> AM to PM
11-02-17 02:12PM <DIR> AM to PM
11-02-17 03:33PM 9276 logo.gif
05-11-20 22:11 <DIR> src
11-02-17 01:23 1 12
11-02-17 4:54 0 icon.bmp
11-02-17 4:54AM 0 icon.gif
11-02-17 4:54PM 0 icon.png
11-02-17 16:54 0 icon.jpg
"""
expected = [
{
"basic": {"is_dir": True, "name": "docs"},
"details": {"modified": 1486778400.0, "type": 1},
"ftp": {
"ls": "11-02-17 02:00AM <DIR> docs"
},
"ftp": {"ls": "11-02-17 02:00AM <DIR> docs"},
},
{
"basic": {"is_dir": True, "name": "images"},
"details": {"modified": 1486822320.0, "type": 1},
"ftp": {
"ls": "11-02-17 02:12PM <DIR> images"
},
"ftp": {"ls": "11-02-17 02:12PM <DIR> images"},
},
{
"basic": {"is_dir": True, "name": "AM to PM"},
"details": {"modified": 1486822320.0, "type": 1},
"ftp": {
"ls": "11-02-17 02:12PM <DIR> AM to PM"
},
"ftp": {"ls": "11-02-17 02:12PM <DIR> AM to PM"},
},
{
"basic": {"is_dir": False, "name": "logo.gif"},
"details": {"modified": 1486827180.0, "size": 9276, "type": 2},
"ftp": {
"ls": "11-02-17 03:33PM 9276 logo.gif"
},
"ftp": {"ls": "11-02-17 03:33PM 9276 logo.gif"},
},
{
"basic": {"is_dir": True, "name": "src"},
"details": {"modified": 1604614260.0, "type": 1},
"ftp": {"ls": "05-11-20 22:11 <DIR> src"},
},
{
"basic": {"is_dir": False, "name": "12"},
"details": {"modified": 1486776180.0, "size": 1, "type": 2},
"ftp": {"ls": "11-02-17 01:23 1 12"},
},
{
"basic": {"is_dir": False, "name": "icon.bmp"},
"details": {"modified": 1486788840.0, "size": 0, "type": 2},
"ftp": {"ls": "11-02-17 4:54 0 icon.bmp"},
},
{
"basic": {"is_dir": False, "name": "icon.gif"},
"details": {"modified": 1486788840.0, "size": 0, "type": 2},
"ftp": {"ls": "11-02-17 4:54AM 0 icon.gif"},
},
{
"basic": {"is_dir": False, "name": "icon.png"},
"details": {"modified": 1486832040.0, "size": 0, "type": 2},
"ftp": {"ls": "11-02-17 4:54PM 0 icon.png"},
},
{
"basic": {"is_dir": False, "name": "icon.jpg"},
"details": {"modified": 1486832040.0, "size": 0, "type": 2},
"ftp": {"ls": "11-02-17 16:54 0 icon.jpg"},
},
]

Expand Down