| clear
query|
facets|
time |
Search criteria: .
Results from 1 to 10 from
48596 (0.0s).
|
|
|
Loading phrases to help you
refine your search...
|
|
[NUTCH-1690] IndexClean: mark url as unindexed after clean to not delete again -
Nutch - [issue]
|
| ...We should marked a deleted page to not delete it again and again. That can simply done by remove Index marker when we delete.I also change to delete duplicated url in solrclean.... |
|
|
|
|
|
|
|
[expand - 2 more]
[collapse]
-
FW: Styles -
Nutch - [mail # user]
|
| ...was not sent to list -----Original message-----> From:Sebastian Nagel > Sent: Saturday 19th August 2017 13:49> To: Markus Jelsma > Subject: Re: Styles> > There are 3 r... |
|
|
|
|
| |
|
Styles - Nutch - [mail # dev]
|
| ...Although it has not troubled me so far, if there is something you think there is to improve i would most certainly welcome it! Feel free to open an issue at our Jira, provide a patch or pull... |
|
|
|
|
|
Styles - Nutch - [mail # dev]
|
| ...I've noticed our imports have inconsistent import orderings, is thereanyone who would like to see this made consistent? I have the time to workon this, if it is important enough to others as... |
|
|
|
|
|
|
|
[expand - 1 more]
[collapse]
-
Sitemap detection bug? -
Nutch - [mail # user]
|
| ...Hi Sebastian,Sorry forgot to reply to list.I remember enabling debug logging once before and found that the parsing of robots.txt stops after it finds the entry relevant to the crawler ID. I... |
|
|
|
|
| |
|
Sitemap detection bug? - Nutch - [mail # user]
|
| ...Hi,I've been unable to detect the sitemap for https://www.mscdirect.com/robots.txt, I did some searching and I think it might be due to their robots.txt line spacing format. I tried user-age... |
|
|
|
|
|
|
|
[NUTCH-2316] Library conflict with Parser-Tika Plugin and Lib Folder -
Nutch - [issue]
|
| ...Hello Apache Nutch Team,everytime Nutch wants to parse a *.class file this Exception pops up:java.util.concurrent.ExecutionException: java.lang.IncompatibleClassChangeError: class org.apache... |
|
|
|
|
|
|
|
[NUTCH-2071] A parser failure on a single document may fail crawling job -
Nutch - [issue]
|
| ...java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357) at org.apache.nutch.parse.ParseSegmen... |
|
|
|
|
|
|
|
[NUTCH-2378] ChildFirst plugin classloader -
Nutch - [issue]
|
| ...While working on upgrading the indexer-elastic plugin from 2.x to 5.x, I ran into several nasty runtime dependency issues (both local and on Hadoop). After seeking help on the mailing list, ... |
|
|
|
|
|
|
|
[expand - 13 more]
[collapse]
-
Best practice for Nutch 2.x on AWS? -
Nutch - [mail # user]
|
| ...Hi Michael,except for HBase and Solr nothing has to be deployed or installed.On the Hadoop master node:- build Nutch via ant runtime- point NUTCH_HOME to the directory where the ... |
|
|
|
|
| |
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Glad you found the solution.On Thu, Aug 17, 2017 at 12:46 PM, Michael Chen <[EMAIL PROTECTED]> wrote:> Fixed the problem... It was most likely a table match problem: it is> neces... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Fixed the problem... It was most likely a table match problem: it is necessary to specify -crawlId during indexing. Also the "Total 0 document is added" is probably a bug... The MR input out... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Hi Divjot,You're right. I checked the webapp and rootdir is already defined by "hbase-site.xml" outside of Nutch, probably by CloudEra, though it is strange why CloudEra didn't take care of ... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Hi Divjot,Thanks for the reply! I checked the HBase tutorial but still am a bit confused. When I set up the standalone build, hbase-site.xml resides in the hbase conf/. But it seems that wit... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Hi Divjot,I have a cluster running with CloudEra Manager (Hadoop, HBase, Solr, ZooKeeper). Do you know if I need to modify the hbase-site.xml before "ant runtime"? What configurations did yo... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...The problem was caused by mismatched JDK versions. I had to use 1.8 to compile Ant and Nutch, but CloudEra Manager defaults to JDK1.6 or 1.7. Solution is to override JAVA_HOME in hosts confi... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Hi,I just figured out how to deploy jobs to hadoop with the jar file... But I ran into an error during the first injection step:java.lang.UnsupportedClassVersionError: org/apache/gora/mapred... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Hi Sebastian,Thanks for the reply. I do have to use 2.x for some functionalities, so I guess I might have to stick to HDFS for now...I set up a 5-node hadoop cluster with HBase and Solr serv... |
|
|
|
|
|
Best practice for Nutch 2.x on AWS? - Nutch - [mail # user]
|
| ...Hi Michael,> Will I be able to use S3 as data storage so that I can keep the data when EC2 instance stops?I don't know whether this is easily possible for 2.x and HBase. But Nutch 1.x can... |
|
|
|
|
|
|
|
Parse Timeout? -
Nutch - [mail # user]
|
| ...Hi,I've been getting a strange timeout exception during parsing of a large sitemap XML document. I've set the timeout in nutch-site.xml to -1 or large numbers and "ant clean && ant runtime" ... |
|
|
|
|
|
|
|
NutchServer -
Nutch - [mail # dev]
|
| ...Quick question about the NutchServer class. The JAXRSServerFactoryBeandocumentation seems to ask for a call to close or destroy on the server itcreates, but I notice the NutchServer close si... |
|
|
|
|
|
|
|
[NUTCH-2410] Unit test for jsoup-extractor not to depend on external resources -
Nutch - [issue]
|
| ...The unit test of the plugin jsoup-extractor depends on a document hosted on youtube.com: private static final String SAMPLE_URL = "https://www.youtube.com/watch?v=pzMpwW4ppRM";It's nic... |
|
|
|
|
|
|
|
|