Packetizer Forums: Pug - Cloud File Archiver2022-03-13T14:07:39ZPacketizer Forumshttps://forums.packetizer.com/webmaster@packetizer.comhttps://forums.packetizer.com/feeds/?f=76Copyright (C) 2022 Packetizer, Inc. All Rights Reserved.Packetizer ATOM/RSS Feed Generatorhttps://www.packetizer.com/rss/images/packetizer.pnghttps://forums.packetizer.com/images/packetizer_icon.pngpaulejPug 1.4 Releasedhttps://forums.packetizer.com/viewtopic.php?f=76&t=1487&p=3005#p30052022-03-13T14:07:39Z2022-03-13T14:07:39Z<div>Pug version 1.4 was released. It is a minor upgrade to the previous 1.3 release. The most notable addition is support for running Pug in containers (e.g., Docker or Podman).<br/>
<br/>
See: <a href="https://pug.packetizer.com">https://pug.packetizer.com</a></div>paulejRe: Doesn't SpiderOak already do all this?https://forums.packetizer.com/viewtopic.php?f=76&t=351&p=1829#p18292016-03-01T22:16:19Z2016-03-01T22:16:19Z<div>Pug will recognize duplicate files and not back up files more than once, but it's not a disk cleanup utility. It's a cloud archiver designed primarily for SMB NAS servers.</div>jgismeRe: Doesn't SpiderOak already do all this?https://forums.packetizer.com/viewtopic.php?f=76&t=351&p=1818#p18182016-02-29T09:58:00Z2016-02-29T09:58:00Z<div>First, I've never heard of either SpiderOak or Pug.<br/>
<br/>
But I do have a problem that isn't all that different from the source maintenance problems of a lot of people, my guess; And I need a solution.<br/>
<br/>
Okay, so I have a really big directory, wide and very deep. With lot's of duplicated and near duplicate files. In fact at one point I had a disk error of some kind (I use an encryption tool, the actual disk platter is the proverbial "bag of bits" and as a consequence of this disk error some elements of the directory tree are repeated maybe a dozen times.<br/>
<br/>
And I want a program to go through and make wise choices, wrt clean-up.</div>paulejRe: Unablt to store archive in cloud storage in ArchiveFilehttps://forums.packetizer.com/viewtopic.php?f=76&t=472&p=1301#p13012014-05-12T07:13:57Z2014-05-12T07:13:57Z<div>I tried to get that version of CentOS installed and working. It was painful. The OS installed, but I could not get Perl modules installed. I could not use yum for some, cpan would not work for me properly, etc. So, I just gave up. I've never encountered so many problems back-to-back like that.<br/>
<br/>
That said, I am wondering if adding the [ ] is a problem. That CentOS you told me you were using has Perl 5.10. I am using Perl 5.14 and the syntax I have in the Amazon S3 related files looks like this:<br/>
<CODE><s>[code]</s>enum 'HTTPMethod' => qw(DELETE GET HEAD PUT);[/code]</CODE>
So, I don't think deprecation is an issue. However, there is a problem with some versions of the Amazon S3 code that requires correction. In the Perl module ListBucket.pm, I had to change these lines as shown:<br/>
<CODE><s>[code]</s><i>
</i>22c22
< foreach my $method qw(prefix delimiter max_keys marker) {
---
> foreach my $method ( qw(prefix delimiter max_keys marker) ) {
[/code]</CODE>
That was the only change I had to make to the S3 Perl libraries and I've used several different versions during testing. The latest S3 code has this fix, but my libraries are a little older.<br/>
<br/>
So, what happens if you do not make the changes as outlined at <a href="http://bitmuncher.subnetworx.de/2014/03/14/passing-list-values-enum-deprecated-netamazons3/">http://bitmuncher.subnetworx.de/2014/03/14/passing-list-values-enum-deprecated-netamazons3/</a>? Do you get the same or a different error?<br/>
<br/>
Paul</div>paulejRe: Unablt to store archive in cloud storage in ArchiveFilehttps://forums.packetizer.com/viewtopic.php?f=76&t=472&p=1298#p12982014-05-09T01:17:37Z2014-05-09T01:17:37Z<div>Samba shouldn't be an issue. I would assume SELinux is also OK, though I do rather hate SELinux and do disable it immediately when I install Linux. But, I'll leave it alone for this. I'll try to test it this weekend.</div>beardwwRe: Unablt to store archive in cloud storage in ArchiveFilehttps://forums.packetizer.com/viewtopic.php?f=76&t=472&p=1297#p12972014-05-08T18:41:58Z2014-05-08T18:41:58Z<div>Also, it should be noted that I'm using SELinux w/ working Samba.</div>beardwwRe: Unablt to store archive in cloud storage in ArchiveFilehttps://forums.packetizer.com/viewtopic.php?f=76&t=472&p=1295#p12952014-05-08T17:12:21Z2014-05-08T17:12:21Z<div>CentOS-6.5-x86_64-bin-DVD1<br/>
<a href="http://isoredirect.centos.org/centos/6/isos/x86_64/CentOS-6.5-x86_64-bin-DVD1.iso"><s>[url]</s>http://isoredirect.centos.org/centos/6/isos/x86_64/CentOS-6.5-x86_64-bin-DVD1.iso[/url]</a><br/>
Some updates I applied to Net :: Amazon :: S3 :: Client :: Object<br/>
<a href="http://bitmuncher.subnetworx.de/2014/03/14/passing-list-values-enum-deprecated-netamazons3/"><s>[url]</s>http://bitmuncher.subnetworx.de/2014/03/14/passing-list-values-enum-deprecated-netamazons3/[/url]</a><br/>
My yum-install list (had to install some pre-req libraries to get some cpan packages to install correctly)<br/>
<a href="https://docs.google.com/document/d/17HOzQP_lRaoxM8vvSJEkL3gulgtyFG1bsIHa5y7m-B4/edit?usp=sharing"><s>[url]</s>https://docs.google.com/document/d/17HOzQP_lRaoxM8vvSJEkL3gulgtyFG1bsIHa5y7m-B4/edit?usp=sharing[/url]</a></div>paulejRe: Unablt to store archive in cloud storage in ArchiveFilehttps://forums.packetizer.com/viewtopic.php?f=76&t=472&p=1292#p12922014-05-08T04:05:47Z2014-05-08T04:05:47Z<div>Can you point me to the exact OS release you're using? I'll try to set up a VM with that image to see why the error is occurring. I might need a few days, but I'm sure we can get it working once I can see the error for myself.</div>beardwwUnablt to store archive in cloud storage in ArchiveFilehttps://forums.packetizer.com/viewtopic.php?f=76&t=472&p=1291#p12912014-05-07T23:56:33Z2014-05-07T23:56:33Z<div>I've been wrestling with getting this working for a week or so now and would love some assistance. I run discover, the archive and this is my output:<br/>
<CODE><s>[code]</s>pug_archiver[3956]: Unable to upload to Amazon S3: Not Implemented at /usr/local/lib/perl5/site_perl/5.18.0/Net/Amazon/S3/Client.pm line 106.
pug_archiver[3956]: Unable to store archive in cloud storage in ArchiveFile
[/code]</CODE>
Any ideas where to go from here? (I've disabled iptables and reran with the same results)<br/>
I couldn't find any error logs in /var/log - any idea where they might be hiding?<br/>
<br/>
CentOS 2.6.32-431.el6.x86_64<br/>
perl 5, version 18, subversion 0 (v5.18.0) built for x86_64-linux<br/>
mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1<br/>
cpan /usr/local/bin/cpan version 1.61 calling Getopt::Std::getopts (version 1.07 [paranoid]),<br/>
<CODE><s>[code]</s><i>
</i> ... snip /usr/local/lib/perl5/site_perl/5.18.0/Net/Amazon/S3/Client.pm
76 sub _send_request_raw {
77 my ( $self, $http_request, $filename ) = @_;
78
79 return $self->s3->ua->request( $http_request, $filename );
80 }
81
82 sub _send_request {
83 my ( $self, $http_request, $filename ) = @_;
84
85 my $http_response = $self->_send_request_raw( $http_request, $filename );
86
87 my $content = $http_response->content;
88 my $content_type = $http_response->content_type;
89 my $code = $http_response->code;
90
91 if ( is_error($code) ) {
92 if ( $content_type eq 'application/xml' ) {
93 my $doc = $self->s3->libxml->parse_string($content);
94 my $xpc = XML::LibXML::XPathContext->new($doc);
95 $xpc->registerNs( 's3',
96 'http://s3.amazonaws.com/doc/2006-03-01/' );
97
98 if ( $xpc->findnodes('/Error') ) {
99 my $code = $xpc->findvalue('/Error/Code');
100 my $message = $xpc->findvalue('/Error/Message');
101 confess("$code: $message");
102 } else {
103 confess status_message($code);
104 }
105 } else {
106 confess status_message($code);
107 }
108 }
109 return $http_response;
110 }
[/code]</CODE></div>paulejRe: Doesn't SpiderOak already do all this?https://forums.packetizer.com/viewtopic.php?f=76&t=351&p=1063#p10632013-10-08T02:50:05Z2013-10-08T02:50:05Z<div>That's different. They're selling a service. I did not want to purchase a service to manage my backups. I want to be in total control of my backup solution.<br/>
<br/>
Pug is just the raw "backup engine". It has a module for storing backups in Amazon S3, but that module could be replaced to work with any cloud storage service or even a corporation's own cloud storage systems.<br/>
<br/>
Pug might work cross-platform, but it was written explicitly for Linux (and other Unix-like systems) and can back up any disk storage that is accessible to the Linux machine. That might be local disk storage or remote storage mounted via CIFS, NFS, or other mountable file system.<br/>
<br/>
Pug is for the IT department that wants to completely manage their backup strategy. The software has lots of nerd knobs for controlling detection of new and modified files, as well as scheduling storage of those files. Pug can use a single database and cloud storage repository for the entire enterprise (or department), or it can be installed such that each installation uses its own database and back-end storage. There are pros and cons with each approach, but the flexibility is there.<br/>
<br/>
I have a dedicated machine that runs Pug and backs up all files from multiple "locations". There's really not a lot of load on the backup server, but it scans mounted network locations, schedules files for archival, and then archives them. Since Pug never stores the same file twice, this approach helps to reduce cloud storage costs when archiving multiple NAS servers and when there are duplicate files. (I see that a lot.)<br/>
<br/>
In any case, Pug is not designed for the typical Windows or Mac consumer. Rather, Pug was designed for the experienced IT system administrator that wants to have total control of cloud archival of corporate data.<br/>
<br/>
Pug could also be turned into a useful service where one sells a physical device with Pug installed on it to business customers. This box would need to have a GUI for the non-technical users, of course. It's a nice solution since one does not install software all over the place to perform backups. Rather, one utilizes an enterprise NAS server and this small device to perform backups. (This would be something exactly like what I use today, though I don't have a GUI.)<br/>
<br/>
I hope that helps.</div>barefootnhDoesn't SpiderOak already do all this?https://forums.packetizer.com/viewtopic.php?f=76&t=351&p=1062#p10622013-10-07T17:27:56Z2013-10-07T17:27:56Z<div>I read through your goals, and re-read them again... Doesn't SpiderOak.com already fulfill all of your bullet points? It does local compression and encryption, de-duplication (strictly confined to only your account), it allows control over its backup scheduling (from immediate to whenever), and unlimited historical versioning. <br/>
<br/>
It's cross-platform: I've been using it under Linux and Windoze for maybe a few years now. I also set it up for a local SMB to backup all their private data that the Feds require them to safely keep.<br/>
<br/>
It will also optionally synchronize directories with an unlimited number of other computers within your account.<br/>
<br/>
It's the only service that handles the privacy issues correctly: local strong encryption where the keys stay with you. (Wuala might be a distant contender.) <br/>
<br/>
Their newest version (5) introduces an optional Dropbox-like feature.<br/>
<br/>
It's not yet open-source.</div>paulejIntroduction to Pughttps://forums.packetizer.com/viewtopic.php?f=76&t=312&p=905#p9052013-01-04T04:19:54Z2013-01-04T04:19:54Z<div><IMG src="http://pug.packetizer.com/images/pug_transparent_small.png"><s>[img]</s><a href="http://pug.packetizer.com/images/pug_transparent_small.png">http://pug.packetizer.com/images/pug_transparent_small.png</a>[/img]</IMG><br/>
<a href="http://pug.packetizer.com/"><s>[url=http://pug.packetizer.com/]</s>Pug[/url]</a> is automated backup software designed for use on Linux that allows one to copy files on a server to cloud storage, compressing and encrypting each file individually before uploading.<br/>
<br/>
In creating Pug, I had several goals:<br/>
<LIST><s>[list]</s>
<LI><s>[*]</s>I wanted a fully automated solution where I didn't have to be bothered with physical storage devices, storing physical drives in a bank vault, etc.</LI>
<LI><s>[*]</s>Once put in place, I wanted a solution that "just worked" and was fault tolerant</LI>
<LI><s>[*]</s>I wanted a solution that was secure</LI>
<LI><s>[*]</s>I wanted a solution that would work incrementally, storing only new or changed files each day (or even each minute)</LI>
<LI><s>[*]</s>I wanted a means of storing multiple versions of files, any one of which I can retrieve</LI>
<LI><s>[*]</s>I wanted a solution that would not waste space by storing the same file multiple times</LI>
<LI><s>[*]</s>I wanted a solution that would preserve locally deleted files for whatever period of time I specify</LI>
<LI><s>[*]</s>I wanted a solution that would allow me to recover any version of a file for some period of time</LI>
<LI><s>[*]</s>I wanted a solution that I could rely on even in the face of a natural disaster</LI>[/list]</LIST>
It was pretty clear that a cloud-based solution was needed, but I could not find any software that would do exactly what I wanted. There were tools that would do synchronization of directories. There were other automated tools that ran on on Windows. There were "cloud backup" programs that were more-or-less like using "tar" and then using s3cmd to shove data into Amazon S3. But, I needed a solution for Linux, since that's where all of my data is located, and I wanted something different than what I could find on the Internet.<br/>
<br/>
So, the idea for Pug was born. The software is so named, because there seems to be general consensus among dog owners that pugs are some of the laziest dogs in the world. Whether that is true or not, I don't know. However, while it was not laziness that brought about the birth of Pug, I'll have to admit that there is an element of laziness or boredom dealing with weekly backups. I wanted something were I could just turn it on and forget about it. Also, Pug is designed to be a little "lazy" about deleting files. (That's configurable, of course, but I prefer that Pug not delete a file for a few months, just in case I change my mind. You can even tell Pug to never delete a file.)<br/>
<br/>
I also needed a very good cloud storage service. For that, the clear winner for me was Amazon S3. Their prices are excellent, as is the service. It's very fast and reliable.<br/>
<br/>
I now have Pug running and everything is fully automated. The software is still "beta", though, as there might still be some bugs that I haven't uncovered. I welcome you to give it a try, though. Start small to see how well it works for you. If it works well, perhaps you can design your backup strategy around it.<br/>
<br/>
The software is open source and free, of course. And by "free", I mean it is free without any restrictions whatsoever. I would even welcome software vendors to utilize the Pug software, database schema, or design ideas and create variants of the software for other platforms, add a GUI front-end, etc. I do work on Linux and my career has been spent on "behind the scenes" stuff, but I'm sure somebody could benefit from similar tools on Windows or a nice user interface. <br/>
<br/>
If you have any questions or comments, please post them to this forum.<br/>
<br/>
Cheers!<br/>
Paul</div>