使用PySpark我试图从elasticsearch读取数据。通常,我会将查询设置为行上的某个内容(请参见下面的查询),并将es.resource设置为索引,例如“my_index/doc”,这样我就可以将数据读入spark:
q ="""{
"query": {
"match_all": {}
}
}"""
Py4JJavaError: An error occurred while calling
z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD.
: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:
org.elasticsearch.hadoop.rest.EsHadoopRemoteException:
invalid_index_name_exception: Invalid index name [_xpack], must not start with '_'.
null
有没有人试过使用xpack或者知道如何从Elasticsearch hadoop插件执行Elasticsearch SQL查询?
q = """{"query": "select * from eg_flight limit 1"}"""
es_read_conf = {
"es.nodes" : "192.168.1.71,192.168.1.72,192.168.1.73",
"es.port" : "9200",
"es.resource" : "_xpack/sql",
"es.query" : q
}
es_rdd = sc.newAPIHadoopRDD(
inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
keyClass="org.apache.hadoop.io.NullWritable",
valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
conf=es_read_conf)